It's Not the Heat, It's the Hubris
Poor reporting and unsupported claims — yes, it's another story about preprints
A story about preprints published last Friday at FiveThirtyEight. Written by Maggie Koerth and entitled, “How Science Moved Beyond Peer Review During the Pandemic,” the reporting turns out to be inaccurate at key points, and Koerth oversimplifies what could have been a more enlightening and complex story.
Koerth’s piece also includes a quote that represents a certain kind of vacuous techno-utopian hubris that has been imported into scholarly publishing.
The inaccuracies occur around how she portrays the publication of results of the RECOVERY trial, with Koerth writing:
. . . the large trial of the steroid dexamethasone — which first demonstrated that this cheap, widely available drug could reduce the likelihood of death in COVID-19 patients — was originally posted as a preprint. It would show up, traditionally published, in the New England Journal of Medicine eight months later, but the preprint made it possible to get that knowledge into doctors’ hands faster.
Koerth’s timeline is wrong. The results were first published in a press release from the University of Oxford on June 16, 2020, in conjunction with the announcement that the trial had been stopped because clinical benefits were accruing. The preprint she mentions was posted nearly a full week later. And the paper in NEJM, which represents the first peer-reviewed and editorially approved version of the research, occurred about three weeks after that, not eight months later.
A final smaller error of omission occurs in Koerth’s reporting in that she fails to note that NEJM also produced a key editorial along with its initial publication of the results, putting them in context for researchers and clinicians. We’ll revisit a bit of this editorial when we talk about the hubris of techno-utopians.
The actual timeline of the RECOVERY trial is more complicated and informative than Koerth may realize. (I reached out to her via Twitter about the mistakes and omissions in her timeline, and had not heard back as of this writing.)
- The RECOVERY Trial itself received ethics approval in the UK March 17, 2020, and began in the UK and US on March 19, 2020. It was stopped June 8, 2020.
- The University of Oxford issues a press release about the results of the RECOVERY trial on June 16, 2020 — no preprint is available for scientific scrutiny
- Author disclosures are first completed using NEJM forms starting on June 8, 2020 — the day the trial was stopped — and more than a week before the Oxford press release, suggesting that the authors were preparing to submit to NEJM more than a week before Oxford issued its press release, and as soon as the trial stopped
- A preprint is published on medRxiv on June 22, 2020 — also the last date on disclosures filed with NEJM, providing some circumstantial evidence that it’s the date the paper was submitted, and that the authors coordinated paper submission and preprint posting — a common practice
- NEJM published its peer-reviewed version of the paper on July 17, 2020, just over three weeks after the preprint was posted and the paper was, I think, officially submitted
Koerth states in her piece with an incredible amount of certainty that “research in preprints isn’t a lot different statistically from how those same papers eventually end up being published in a peer-reviewed scientific journal.” To back this up, she cites a very weak study.
But let’s examine this question in this instance — What did the NEJM version of the report include that the preprint did not?
- more patient outcomes
- refined and expanded statistical analyses
- refined and expanded methodology descriptions
- more precise language around statistical methods and inferences
- a more robust overview data table
- a randomization schematic absent from the preprint
- a table of primary and secondary outcomes absent from the preprint
- an accompanying editorial co-authored by Clifford Lane and Anthony Fauci
NEJM published a final version of the paper and editorial on February 25, 2021, in a print issue, and this version is now the online version, with the date being what Koerth used to imply that NEJM took “eight months” to publish the results. This is the kind of error common to tourists.
As noted above, we can assume the paper was submitted to NEJM on June 22, 2020, as that’s the date on the final disclosure form, which came from Martin Landray, the last author listed on the preprint and the putative PI for the group. As the NEJM version tells us (but not the preprint), he and the first author (Horby) drafted the paper and vouched for it. It’s likely Landray collected the disclosures for the other authors in the writing collaborative, and then submitted them with his own on June 22, along with the paper.
If NEJM received the submission on June 22, the date the preprint was posted, that means NEJM published the work in 20 workdays, not 8 months. And we don’t know what led to the delay, because NEJM (along with other journals these days) can turn papers around quickly, with peer review, statistical review, additional data, multiple revisions, and so forth. Were the authors swamped with media requests? Were they slow to respond to additional data requests? Or did the rapid uptake of the news from Oxford blunt the need for NEJM or the authors to rush?
Note that it took Oxford a week+ after the trial stopped to issue its press release, and the authors two weeks from same to post a preprint.
The final version of the NEJM paper includes:
- a “Quick Take” for clinicians
- an expanded overview table, including data by race
- an expanded table of primary and secondary outcomes
Might it be worthwhile celebrating a journal sticking with authors for eight months to get as much data and information to clinicians and other researchers as possible? The preprint server just stuck a paper up after a simple screening process, and moved on. Yet Koerth makes it sound like they did something honorable and revolutionary — because that’s the working narrative of techno-utopianism.
And so we get to the next item in the mythology of the power of tech . . .
The Techno-Utopian as Hero
There was to me one especially galling aspect of Koerth’s story, which arrives via a quote from Richard Sever of bio/medRxiv:
“I had one MD who contacted me, and he said, ‘You know, there are probably people who are alive today who would have been dead if not for preprints.’”
Ah, the techno-utopian as hero.
This statement is repulsive and inappropriate.
I’ve been involved in medical publishing for decades, and worked on various editorial, publishing, and policy aspects of papers involving infectious disease, tropical disease, neurological disease, emergency medicine and trauma, pediatric disease, and so forth, at a fairly high rate and a fairly high level. Even when journals I was involved with published papers that helped reduce infant deaths by a huge amount, or led to the recall of dangerous drugs or devices, or trumpeted the discovery of amazing cancer therapies, I never — not once — believed the people working to examine, refine, and disseminate papers deserved the reflected glory of those achievements. We supported and helped researchers reach the right audience with the most accurate information possible. That’s noble work when you put your heart into it, but it’s not heroic, and certainly nothing approaching saving a life. And that’s how Sever’s statement struck me — as someone who has done very little (far less than a typical publishing professional, actually), and then attempts to collect massive and undeserved credit.
Claiming that pushing a raw manuscript out on a preprint server was a life-saving act is vacuous techno-utopian self-congratulation at its finest.
Let’s look at what more likely happened with the RECOVERY trial in particular. Physicians in multiple centers in the UK’s NHS, via their hard-earned expertise, years of experience using such steroids to help ameliorate the symptoms of SARS, MERS, and influenza, and their rapid assessment of the clinical features of severe Covid-19 infection, determined that adding dexamethasone was a low-risk and possibly helpful intervention that fit with prior clinical experience and research. They tried it, and saw some intriguing results that warranted further study. They put together a collaborative, multi-center trial to rapidly study the intervention’s effects on as large a population as feasible, and carefully tracked the data and outcomes. In a few months, the study was stopped because they did see a positive effect. Stopping the trial led Oxford to release some preliminary results in a press release just over a week later to explain what was happening.
No preprint yet, but certainly a bunch of great doctors and clinical staff to congratulate. Thank the heavens for them.
The preprint went up about a week later. By the time it was up, the news media had saturated the electromagnetic spectrum with coverage of the press release, which was triggered by the stoppage of the trial. A full week passed between news coverage and preprint. At Internet pace, that’s a lifetime. The preprint was already being left in the dust by then.
Even if the preprint helped guide some level of clinical decision-making, that’s small potatoes compared to the thousands of doctors and nurses and respiratory therapists and surgeons and clinical staff who intubated, treated, and monitored hundreds of thousands of patients around the world in grueling shifts that we never had to really witness, except through occasional sanitized television news coverage. And it’s certainly not anything any one of us should attempt to gain credit from.
Even if you accept the premise that preprints had an effect on the pandemic, the jury is definitely out on the net effect of the noisy, tumultuous, unmitigated information space they contributed to. The most common type of Covid preprint — attempts to apply AI to the disease — generated no meaningful clinical results. Other preprints misled patients, nurses, and physicians, and we continue to see the ill-effects of conspiracy theories around the origin of the disease and the effects of vaccines.
If you’re going to attempt to claim some credit, you have to be willing to take some blame, too. And there have been plenty of problems stemming from Covid-19 preprints, including some retracted from bioRxiv and medRxiv. But Sever has never to my knowledge mentioned these gaffs or the overall low yield of preprints. And he doesn’t seem to want to contemplate any possible culpability for even one of the more than 400,000 excess deaths in the US alone during the pandemic.
Saying “preprints saved lives” also diminishes the role of patients and their families during times like these, as the editorial writers at NEJM noted:
The key to the success of the RECOVERY trial has been its pace of enrollment. The ability to rapidly enroll thousands of patients into the trial no doubt was facilitated by the National Health Service in the United Kingdom and the fact that the trial was available to essentially the entire patient population of the country.
Let’s retain some humility about our role in science and medicine. We didn’t have to wash viscera off our hands. We didn’t have cleanse and salve bed sores. We didn’t have to call a code or declare a patient dead. We didn’t have to wear a mask 12 hours straight day after day. And when it comes to preprints, Sever and his colleagues can’t even say they tried to make sure clinicians and scientists received the best and most complete information possible. They just approved draft papers that passed some basic sniff test — and that made them feel like heroes, I guess.
This brand of techno-utopian hubris is our space’s equivalent of whatever this was supposed to be:
