Maximizing Benefit, Minimizing Harm

Last week, I critiqued an assessment in Nature of quantitative publishing trends during Covid-19 — a critique that I’m grateful led to both a clarification to the original piece and a nice exchange with Richard Van Noorden, in which I asserted that perhaps the story isn’t a quantitative one, but rather a qualitative one. That is, has the haste and volume in publishing tens of thousands of items helped or harmed science and society?

Little did I know that not 24 hours later, I’d be reading an in-depth story teasing out some of these same ideas by Ed Yong at the Atlantic. More on this below.

While viable vaccines are great news, and milestones in biomedical science built on mRNA research that began 30 years ago, these wins are offset by devastating losses, largely due to an over-emphasis on science while addressable and proven public health measures were sidelined. Last Friday, the US documented more than 220,000 new cases of Covid-19 in a single day. With a case-fatality rate of around 2%, in a few weeks we’ll likely see 4,500+ deaths from Friday‘s cases. Health care workers are exhausted. A person is dying of Covid-19 every 15 seconds in Los Angeles county. And so on. It’s a litany of premature and preventable death on a mind-numbing scale.

It didn’t have to be this way, vaccine or no.

This year, scientists published more papers and posted more preprints about Covid-19 than they have ever on any topic in a similar timeframe. NEJM alone reportedly received 16,000 excess submissions, all Covid-related. Anecdotes I’ve gathered suggest thousands of papers were circulating during the spring from publisher to publisher, bouncing around and consuming time, attention, and bandwidth in dozens or hundreds of editorial offices, while resulting in a relatively low ratio of useful publication events.

All of this science activity — heat, rather than light — may have contributed to a public gestalt of “science will find something unexpected,” subsuming and sidelining voices from the public health space. For months, there were so many confusing, untested claims circulating — all bursting with hope and the emotional manipulation of press releases — that basic and proven public health measures like masks, physical distancing, and quarantine felt like afterthoughts. The feeling seemed to be, “Don’t worry, this won’t last long — the silver bullet will arrive any day.”

Now that a couple of silver bullets are chambered, there are still concerns that we’ll be able to aim and fire effectively enough to turn vaccines into vaccinations. Even if we’re efficient and effective, rebuilding from this point will take time, effort, and patience.

We can use this time to take stock of whether the benefit of all these papers being pushed out in all their myriad forms outweighed the costs and potential harms. And to ask how we can build on the benefits and shed the harmful aspects.

More and more scientists and information professionals are growing worried about what the Covid-19 publishing environment says about our ability to respond efficiently and effectively during a crisis. Madhukar Pai from McGill University wrote in Nature back in July:

We have already seen a flood  of dubious and retracted research, and a lowering of normal scientific standards. Bad science, combined with poor science communication, is eroding public faith in research and is leading policymakers astray.

Yong’s article in the Atlantic about the Covid-19 pandemic focuses on vaccine distribution, but he tackles some of the problems that have emerged with science communication, emphasizing the pros and cons of rapidly produced information — and its uneven reliability — in particular:

These efforts have already paid off. New diagnostic tests can detect the virus within minutes. Massive open data sets of viral genomes and Covid‑19 cases have produced the most detailed picture yet of a new disease’s evolution. Vaccines are being developed with record-breaking speed. SARS‑CoV‑2 will be one of the most thoroughly characterized of all pathogens, and the secrets it yields will deepen our understanding of other viruses, leaving the world better prepared to face the next pandemic.

But the Covid‑19 pivot has also revealed the all-too-human frailties of the scientific enterprise. Flawed research made the pandemic more confusing, influencing misguided policies. Clinicians wasted millions of dollars on trials that were so sloppy as to be pointless. Overconfident poseurs published misleading work on topics in which they had no expertise. Racial and gender inequalities in the scientific field widened.

. . . when people look back on this period, decades from now, they will also tell stories, both good and bad, about this extraordinary moment for science. At its best, science is a self-correcting march toward greater knowledge for the betterment of humanity. At its worst, it is a self-interested pursuit of greater prestige at the cost of truth and rigor. The pandemic brought both aspects to the fore. Humanity will benefit from the products of the Covid‑19 pivot. Science itself will too, if it learns from the experience.

This last part is critical, and is something I’ve been advocating for a long time — in fact, it’s kind of the point of this newsletter. We can learn from our mistakes if we’re willing to acknowledge them. We can take critiques offered in good faith, weigh them, and use the valid points to create a better next version, next strategy, next step.

Interviewed on “Fresh Air” last week, Yong puts a finer point on what I think is a current but unnecessary trade-off in the benefits of high-volume, low-scrutiny publishing and its potentially very high cost:

In recent years, biomedical scientists have been making use of preprint servers, places where researchers can put up early versions of their papers before they’ve been reviewed by others, so that their results can be discussed and built upon. Now, this is a development that has both good and bad sides to it. The good side is that it allows science to operate at a much faster pace, and we saw that in the pandemic. We saw preprints being openly and rapidly discussed and dissected, built upon.

But, at a time when a lot of overconfident and inexperienced people are diving into the field, you see a lot of bad work, a lot of sloppy preprints, and a lot of those got massive media attention, because we journalists were looking for them to a greater degree than we’d ever done before. There are both good and bad sides to this, I think. If you want science to operate at a much faster pace than it normally does, then you’re going to see both good and bad work getting out into the world at a much faster pace.

What did we learn from all of these preprints and papers? Not much, it seems, as Yong outlines the problem of “epistemic trespassing”:

. . . [experts] swerving out of their scholarly lanes and plowing into unfamiliar territory. . . . [many] epistemic trespassers spent their time reinventing the wheel. One new study, published in NEJM, used lasers to show that when people speak, they release aerosols. But as the authors themselves note, the same result — sans lasers — was  published in 1946, [Linsey] Marr[, the Charles P. Lunsford Professor of Civil and Environmental Engineering at Virginia Tech,] says. I asked her whether any papers from the 2020 batch had taught her something new. After an uncomfortably long pause, she mentioned just one.

Editors ran with scissors many times during the early days of Covid-19, with some infamously wounded when they stumbled. But preprints have been a wholly different category of problem. As you know, I think making preprints openly available to the lay public and journalists perpetuates unnecessary downsides and mismanages our interface with society. But is there a better way? Criticism is one thing, but what about a positive proposal for fixing the issues? I’ve proposed numerous design improvements before, and here’s a bit more refinement — if preprint servers required a simple email login, and required these emails to be associated with institutions of higher education and research companies, we’d have a much better system for sharing preliminary reports. Not possible? Here’s something Substack has created, a control it gives its writes around special offers. They’ve assured me their list of educational domains is comprehensive:

Refining the access model around biomedical preprints doesn’t seem like a matter of what’s possible. It’s a matter of will and skill.

It’s also worth noting that Yong assumes journalistic norms in his reporting of the possible downsides of a looser publication environment. Unfortunately, those without such norms — the alt-right and far-right media sources like Breitbard, Daily Caller, and Sputnik News — have become quite influential and sophisticated, and appear to be over-sampling preprints in order to mislead and confuse their audiences. This phenomenon has been critical to the shoddy pandemic response, as these audiences overlap significantly with anti-mask movements, anti-vax movements, and legislative and political bodies that are striking down measures to stop the spread in multiple US jurisdictions.

Of course, the weaknesses and downsides include predatory publishers, and the tendency for some business models and broader pressures within scholarly publishing to make us “self-flooding.” The misogyny of scientific culture (and our culture generally) may have also been exposed again, with one preprint suggesting submissions have gone up for scientists — except for female scientists. Many areas of field research and unrelated research have been put on hold, which will create holes in data sets and non-Covid studies which may prove impossible to fill.

The bigger picture of the science and information spaces reveals a system under exquisite pressure:

• A decades-long focus on STEM education has created many trained STEM scientists, but without creating the commensurate career opportunities, fostering a sense of desperation to gain notoriety and funding
• A 40-year deprivation of library budgets after the Reagan Administration decoupled grant funding from library budget growth, creating pressures on libraries and publishers that have led to tension and finger-pointing, and then to business models that opened the door to predation and conflicts of interest
• An attention economy which trades on clicks, likes, and up votes, which has misled some information purveyors — first, newspapers, and then others — into strategies predicated on the notion that advertising and attention naturally accrue into revenues or benefit to users, when in fact an advertising duopoly (Facebook, Google) and abuse of users has resulted
• A science culture that has not adequately addressed the disparities that keep science from being as strong and diverse as it could be

A critical interface for science is the one with the public. We’ve been lax and cavalier about this interface for too long, and some simple changes could greatly improve it — both in the short-term and in case we ever face another global crisis. And while science which started 30 years ago has finally produced the hoped-for vaccine platform right when we needed it, it’s unclear — and even unlikely — that most of the research flurry produced since January 2020 will yield much.

Our best bet is to learn from the mistakes and missteps, and to improve what we can — which, I think, is quite a lot. Or, as Yong concludes:

The scientific community spent the pre-pandemic years designing faster ways of doing experiments, sharing data, and developing vaccines, allowing it to mobilize quickly when Covid‑19 emerged. Its goal now should be to address its many lingering weaknesses. Warped incentives, wasteful practices, overconfidence, inequality, a biomedical bias — Covid‑19 has exposed them all. And in doing so, it offers the world of science a chance to practice one of its most important qualities: self-correction.

Subscribe now

Give a gift subscription