Why you can't compare Covid-19 vaccines

What a vaccine's "efficacy rate" actually means.

Sign up for our newsletter: http://www.vox.com/video-newsletter

In the US, the first two available Covid-19 vaccines were the ones from Pfizer/BioNTech and Moderna. Both vaccines have very high "efficacy rates," of around 95%. But the third vaccine introduced in the US, from Johnson & Johnson, has a considerably lower efficacy rate: just 66%.

Look at those numbers next to each other, and it's natural to conclude that one of them is considerably worse. Why settle for 66% when you can have 95%? But that isn't the right way to understand a vaccine's efficacy rate, or even to understand what a vaccine does. And public health experts say that if you really want to know which vaccine is the best one, efficacy isn't actually the most important number at all.

Further reading from Vox:

Why comparing Covid-19 vaccine efficacy numbers can be misleading: https://www.vox.com/22311625/covid-19-vaccine-efficacy-johnson-moderna-pfizer

The vaccine metric that matters more than efficacy: https://www.vox.com/22273502/covid-vaccines-pfizer-moderna-johnson-astrazeneca-efficacy-deaths

The limits of what vaccine efficacy numbers can tell us: https://www.vox.com/21575420/oxford-moderna-pfizer-covid-19-vaccine-trial-biontech-astrazeneca-results

Vox.com is a news website that helps you cut through the noise and understand what's really driving the events in the headlines. Check out http://www.vox.com.

Watch our full video catalog: http://goo.gl/IZONyE
Follow Vox on Facebook: http://goo.gl/U2g06o
Or Twitter: http://goo.gl/XFrZ5H

This is the new one-dose Covid-19 vaccine
from Johnson & Johnson.

In early March,
more than 6,000 doses

were supposed to be shipped
to the city of Detroit, Michigan

but the mayor said, "No thanks".

"Moderna and Pfizer are the best.

And I am going to do
everything I can

to make sure the residents
of the city of Detroit get the best."

He was referring to these numbers:
the vaccines’ "efficacy rates."

The vaccines from Pfizer/BioNTech
and Moderna

have super high efficacy rates:

95% and 94%.

But Johnson & Johnson?

Just 66%.

And if you only look at these numbers,
it’s natural to think that these vaccines

are worse than these.

But that assumption is wrong.

These numbers are arguably
not even the most important measure

of how effective these vaccines are.

To understand what is,

you first have to understand
what vaccines are even supposed to do.

A vaccine’s efficacy rate
is calculated in large clinical trials,

when the vaccine is tested
on tens of thousands of people.

Those people are broken into two groups:

half get the vaccine,
and half get a placebo.

Then, they’re sent out
to live their lives,

while scientists monitor whether or not
they get Covid-19 over several months.

In the trial for Pfizer/BioNTech, for
example, there were 43,000 participants.

In the end, 170 people
were infected with Covid-19.

And how those people fall
into each of these groups

determines a vaccine’s efficacy.

If the 170 were evenly split,

that would mean you’re just as likely
to get sick with the vaccine

as without it.

So it would have a 0% efficacy.

If all 170 were in the placebo group, and
zero people who got the vaccine were sick,

the vaccine would have
an efficacy of 100%.

With this particular trial,
there were 162 in the placebo group,

and just eight
in the vaccine group.

It means those who had the vaccine
were 95% less likely to get Covid-19:

The vaccine had a 95% efficacy.

Now, this doesn’t mean that if 100 people
are vaccinated, 5 of them will get sick.

Instead, that 95% number
applies to the individual.

So each vaccinated person

is 95% less likely than a person
without a vaccine to get sick

each time they’re exposed to Covid-19.

And every vaccine’s efficacy rate
is calculated in the same way.

But each vaccine’s trial might be done in
very different circumstances.

So, one of the biggest
considerations here,

when we look at these numbers,

is the timing in which
these clinical trials were performed.

This is the number of daily Covid-19 cases
in the US since the pandemic began.

The Moderna trial was done
completely in the US, here, in the summer.

The Pfizer/BioNTech trial was primarily
based in the US, too,

and at the same time.

Johnson & Johnson, however,
held their US trial at this time,

when there were more opportunities

for participants
to be exposed to infections.

And most of their trial took place
in other countries,

primarily South Africa and Brazil.

And in these other countries,
not only were case rates high,

but the virus itself was different.

The trials took place
as variants of Covid-19 emerged,

and became the dominant infections
in these countries;

variants that are more likely
to get participants sick.

In South Africa, most of the cases in the
Johnson & Johnson trial were that of the

variant, not the original strain that was
in the US over the summer.

And despite that, it still
significantly reduced infections.

"If you're trying to make one-to-one
comparisons between vaccines,

they need to have been studied in the same
trial, with identical inclusion criteria,

in the same parts of the world,
at the same time."

"If we were to take
Pfizer and Moderna's vaccines and

redo their clinical trial at the same time
that we saw J&J's clinical trial,

we might see quite different
efficacy numbers for those."

These efficacy numbers just tell you
what happened in each vaccine’s trial,

not exactly what will happen
in the real world.

But many experts argue this isn’t even the
best number to judge a vaccine by anyway.

Because preventing any infection at all is
not always the point of a vaccine.

"The goal of a Covid-19 vaccine program
is not really to get to 'Covid zero',

but it's to tame this virus, to defang it,

to remove its ability to cause serious
disease, hospitalization, and death."

It helps to look at the different outcomes
of an exposure to Covid-19 like this:

The best-case scenario
is, you don’t get sick at all.

The worst case is death.

In between, there’s being hospitalized,
severe-to-moderate symptoms,

or having no symptoms at all.

In the best circumstances, vaccines
give you protection all the way to here.

But realistically, that isn’t the
main objective of the Covid-19 vaccines.

The real purpose is to give your body enough
protection to cover these possibilities,

so if you do get an infection,
it feels more like a cold

than something you'd be
hospitalized for.

And this is one thing that every one
of these Covid-19 vaccines do well.

In all these trials, while some people in
the placebo groups were hospitalized,

or even died from Covid-19,

not one fully vaccinated person,
in any of these trials,

was hospitalized or died from Covid-19.

"One thing I wish that mayor
would have understood

was that all three vaccines have essentially
100% effective in protecting from death."

The mayor of Detroit did backtracked and said
he’d start taking Johnson & Johnson doses,

because it’s still "highly effective
against what we care about most."

Efficacy matters.
But it doesn’t matter the most.

The question isn’t which vaccine will
protect you from any Covid infection,

but which one will keep you alive?

Or out of the hospital?

Which one will help end the pandemic?

And that’s any of them.

"The best vaccine right now for you
is the one that you're offered."

"With each shot that goes into someone arms,
we get closer to the end of this pandemic."