The makers of a third coronavirus vaccine announced positive results in clinical trials on Monday, setting off yet another round of excited news reports. This one, produced by a partnership between a University of Oxford research institute, its spinout company Vaccitech, and the pharmaceutical company AstraZeneca, does not need to be stored at freezing temperatures and would be cheaper and easier to produce than the high-efficacy vaccines produced by BioNTech-Pfizer and Moderna. Indeed, according to an initial write-up in The New York Times, Oxford-AstraZeneca’s is “expected to be relied upon heavily across the globe, to help curb a pandemic that has killed more than 1.3 million people.”

Sounds like great news, right? Monday’s press release from AstraZeneca presents “convincing evidence that [the vaccine] works,” said Science. But not everyone has been convinced. The price of AstraZeneca’s shares actually dropped on the news, and an analysis from an investment bank concluded, “We believe that this product will never be licensed in the US.” Over at STAT News, Anthony Fauci cautioned that we’ll need to see more data before coming to a conclusion. The skeptics have strong reasons to be concerned: This week’s “promising” results are nothing like the others that we’ve been hearing about in November—and the claims that have been drawn from them are based on very shaky science.

SUBSCRIBE

Image may contain Rug

Subscribe to WIRED and stay smart with more of your favorite Ideas writers.

The problems start with the fact that Monday’s announcement did not present results from a single, large-scale, Phase 3 clinical trial, as was the case for earlier bulletins about the BNT-Pfizer and Moderna vaccines. Instead, Oxford-AstraZeneca’s data came out of two separate studies: one in the UK that began in May, and another in Brazil, which got started at the end of June. These two studies were substantially different from one another: They didn’t have standardized dosing schemes across the trials, for one thing, nor did they provide the same “control” injections to volunteers who were not getting the experimental Covid vaccine. The fact that they may have had to combine data from two trials in order to get a strong enough result raises the first red flag.

Consider that leading vaccine makers—including AstraZeneca—issued a scientific-rigor-and-integrity pledge back in September, in which they promised to submit their products for approval or emergency use authorization only “after demonstrating safety and efficacy through a Phase 3 clinical study that is designed and conducted to meet requirements of expert regulatory authorities such as FDA.” Note the wording here: These companies did not suggest that they might claim to have demonstrated efficacy through multiple, distinct clinical studies, combined together to get enough data. They said they would use a Phase 3 study—as in, one big one. Yet AstraZeneca has already applied on the basis of this data for approval in Canada, and has plans to do the same in Britain, Europe and Brazil. The company also says it will use the data to apply for emergency use authorization in the US.

The Food and Drug Administration’s guidance for Covid-19 vaccines does allow for emergency use authorization based on interim analyses, but the same document says this must be supported by a minimum level of vaccine efficacy “for a placebo-controlled efficacy trial.” Again: it refers to a trial. That is exactly what BNT-Pfizer and Moderna did. Both released the FDA-approved blueprints for their trials—called trial protocols—weeks ahead of time, with details of the calculations and statistical rules that they’d use to determine when to perform an interim analysis and how much certainty could be attached to those results. When BNT-Pfizer’s discussions with the FDA led to changes in this plan, BNT-Pfizer explained why, and released an updated protocol. That’s scientific rigor, and it matters a lot. When a vaccine-maker specifies the rules of the game before the results start coming in, we can check their work and be confident in what they tell us at the end. We can make sure they haven’t cherry-picked the data.

The Oxford-AstraZeneca story is very different, though. Presumably, neither of the two trials from which they combined data could have provided a clear answer on the vaccine’s efficacy on its own. To make things worse, Oxford-AstraZeneca reported only the results for certain subgroups of people within each one. (For perspective on this: The two subgroups chosen leave out perhaps half the people in the Brazilian trial.) Meanwhile, one of their key claims is that giving half a dose of the vaccine on the first injection, followed by a standard dose on the second one, led to better outcomes—but neither of these trials had been designed to test this hypothesis. In fact, it’s since emerged that the half-dose/full-dose option started out as a mistake, and one that was only caught when some people in the study didn’t have the usual high rate of adverse effects.