Is It Now Risky to Name Peer Reviewers?

If 2007 was a watershed year for the availability of digital information — with the Kindle, iPhone, Netflix streaming, Android, and Twitter launching or hitting their growth curve around this time — 2016 looks to be another inflection point for digital information. Let’s call it the point at which the spell was broken, and caution and common sense re-emerged.

Accordingly, the role of curators, editors, and referees online has been a hot topic. Bullying and abuse ran rampant for a time, and rates of suicide, depression, and stress skyrocketed as a result. The Internet moved beyond benign to often feeling malign. It needed more referees.

I recently wrote about Michael Lewis’ new podcast, “Against the Rules,” which is focused on the decline of the human referee in many parts of daily life, and what that’s doing to our idea of fairness.

A recent episode about how judges can subtly be influenced and intimidated made me think that perhaps there are underappreciated risks to identifying peer reviewers in an effort to make peer review more “open” or “transparent.” Maybe in the current online world, we need to pause and be more cautious here, too.

It’s not as if open peer review is popular or widespread. When Elsevier piloted open peer review, only 6% of reviewers opted to be identified by name. The EMBO Journal and eLife offer it, as does BMJ Open. It’s relatively rare. But there may be some reasons not to push for it at all.

One thing that Lewis observes repeatedly across many types of judging, refereeing, and regulating is how “the crowd” is now involved. This is a structural change. Attending a major sporting event allows everyone to referee — to watch the slow-motion replay of a disputed call on the Jumbotron, and if you think some call was wrong, you can go to social media to express your indignation, all the while berating the referees.

“The crowd” — and a structurally amplified crowd — is now a factor in refereeing of nearly all kinds. Twitter, cameras, and online search tools have become conveniences for bullies, and there is a heightened level of risk around refereeing thanks to such changes. Referees for most major sports now need bodyguards.

Peer review is part of our refereeing system, and when it comes to open peer review, the idea of “the crowd” came up recently in a post examining some psychological aspects of open peer review, with a commentator noting how being identified went:

I was asked to review a review paper that was in one of those online discussion journals. Because the authors would figure out who I was, I chose to be not anonymous. I thought the review paper was worthwhile, but there were many flaws in the content and writing that required extensive revisions. If done right, this would probably occupy many months of the authors’ time, so I recommended rejection and resubmission after all the flaws were fixed. Friends and colleagues of the coauthors starting posting reviews about how important this paper was to publish, most neglecting my point that it was poorly written and needing substantial work. In this case, the open peer-review process allowed a camp to form around support for the manuscript.

Lewis’ podcast examining the ways judges can be intimidated starts at the extreme end of the spectrum, with authoritarians who expect judges to toe the party line, and who will punish them — with prison, or worse — if they rule against the interests of the leader.

From this clear extreme, things get more subtle, in ways that suggest transparency may lead to intimidation or fear for those placed in a position to render life-altering decisions.

Since 2016, the idea that “open” systems thwart dictators and authoritarians has been turned on its head. We now live in the upside-down, where “flooding the zone” on free-to-use platforms is part and parcel of the rise of authoritarianism online and in the real world. It’s no coincidence that the first US President to actively and personally leverage social media also exhibits strong authoritarian tendencies. As Kara Swisher wrote in the New York Times:

. . . it has become clear to me with every incident that the greatest experiment in human interaction in the history of the world continues to fail in ever more dangerous ways.

As to the point about dictators, Swisher said separately during a podcast:

Dictators love social media, because they take it over. . . . They don’t take it down. They abuse it. They use it.

Naming and identifying referees and reviewers opens the door to similar abuses (and uses) of social media. More insidiously, by putting targets on individuals, we makes individuals aware they are susceptible to targeting. This likely has a chilling psychological effect, which is perhaps why 94% of Elsevier’s peer reviewers prefer to remain anonymous, and why double-blind peer review is trending.

Anonymity was once the norm for reporting on the court system. Judicial decisions used to be described as coming from “the court,” as in “the court ruled.” For a variety of reasons, often in the name of transparency and accountability, things in courtrooms changed — television cameras and sketch artists began to appear, for example — and judges started to be named in news coverage.

Many of us remember Lance Ito from the O.J. Simpson trial. It was a short trip from identity to celebrity for judges. Reporting on Supreme Court rulings is often as political as anything, with personalities seeping through all the time (Did Justice Roberts buck the conservative majority? Did Justice Thomas speak? How are Justice Ginsburg’s ribs? Did you see the movie about her?)

Judges, when named, can become targets. Death threats against Judge Ito were documented at the time. Justices Ginsburg and O’Connor received death threats. In 2016, the Guardian reported that 100 judges and magistrates had received death threats in the preceding 5 years. In 2018, the US Marshals Service — which protects members of the Federal judiciary — recorded more than 4,500 “threats and inappropriate communications” aimed at member of the bench.

Fast-forward to more recent history, and we have a President who has called out judges by name, smiled through calls of “Lock them up!” in reference to legitimate law enforcement officials, and openly intimidated witnesses.

Not surprisingly, threats against member of the Federal judiciary have jumped sharply since Trump became President:

We are now in an environment that isn’t generally supportive of referees, judges, reviewers, umpires, ombudsmen, and arbitrators. There is something systemically wrong with how we treat and respond to referees and judges. We may be more susceptible to adopting the bullying and intimidation habits that have been baked into our lives by social media, believing that these can give us some sort of anonymous leverage.

As a peer-reviewer myself, I appreciate being anonymous. I occasionally review papers submitted by people I know, or who I get to know later. Anonymity spares us both any awkwardness in the future. I also like to know that I won’t be attacked on Twitter in some way because of a review.

In a prestige and reputation economy like ours, people can take revenge in non-obvious ways. Everyone knows this, so being identified is a way of being exposed, which can have a subtle chilling effect. Maybe a reviewer fears she won’t get promoted later because a dean’s feathers were ruffled. Maybe an editor worries he won’t receive a grant if he dings a paper from a lab he might someday want to work for. Maybe a reviewer is worried he won’t get invited to submit a poster at the next annual meeting. FOMO is real, and exposing someone’s identity when they serve as a referee may cause the reviewer to worry more about the social implications of their role than the intellectual aspects of the work. And worrying about real or imagined penalties during an honest review does not make the review better.

Then there is the question of, Who benefits from naming peer-reviewers? What does it matter? Does it make the comments easier to dismiss? Does it make the authors feel like they have some leverage in the relationship? Does it improve reviewer behavior somehow?

To this last point, a “kinder and gentler” review may betray an intimidation effect at work. If reviewers smooth the edges of reviews because they’ll be identified by name, do the authors benefit in any way beyond emotionally? What if important feedback is squelched out of a fear of reprisals?

It turns out there are things that do make referees better — training, diversity, and feedback. If you train people to deal with implicit bias, if you broaden the base of referees, and if you give them feedback, they get better. NBA officials are better than ever — more objective, less likely to miss calls — thanks to the league improving in these areas.

But these improvements have not made life easier for referees. In fact, quite the opposite. Being more objective and less biased has upset fans and superstars alike, as there is no longer much home court advantage or superstar indulgence. The game is fairer, thanks to training, diversity, and feedback. But the fans and players are angrier, and the refs in a more perilous situation because it’s easier to harass them.

Maybe a key element of making judges more willing and able to do their difficult jobs — whether editing and reviewing manuscripts, umpiring baseball games, or correcting grammar — is anonymity. Depersonalized decisions that reflect a valid process may be the best option. Naming names in the current environment seems to place a target on the backs of people who are just trying to be fair.

And that doesn’t seem fair.

Subscribers gain access to more than 100 archival posts and a year of full access to the latest from “The Geyser.” Subscribe today!

Subscribe now

Give a gift subscription