WEBVTT

99:59:59.999 --> 99:59:59.999
One simple vitamin can reduce your risk of heart disease.

99:59:59.999 --> 99:59:59.999
Eating chocolate reduces stress in students.

99:59:59.999 --> 99:59:59.999
New drug prolongs lives of patients with rare disease.

99:59:59.999 --> 99:59:59.999
Health headlines like these are published
every day,

99:59:59.999 --> 99:59:59.999
sometimes making opposite claims 
from each other.

99:59:59.999 --> 99:59:59.999
There can be a disconnect between broad,

99:59:59.999 --> 99:59:59.999
attention-grabbing headlines 
and the often specific,

99:59:59.999 --> 99:59:59.999
incremental results of the medical 
research they cover.

99:59:59.999 --> 99:59:59.999
So how can you avoid being 
misled by grabby headlines?

99:59:59.999 --> 99:59:59.999
The best way to assess a headline’s 
credibility

99:59:59.999 --> 99:59:59.999
is to look at the original 
research it reports on.

99:59:59.999 --> 99:59:59.999
We’ve come up with a hypothetical research
scenario

99:59:59.999 --> 99:59:59.999
for each of these three headlines.

99:59:59.999 --> 99:59:59.999
Keep watching for the explanation
of the first example;

99:59:59.999 --> 99:59:59.999
then pause at the headline to 
answer the question.

99:59:59.999 --> 99:59:59.999
These are simplified scenarios.

99:59:59.999 --> 99:59:59.999
A real study would detail many more 
factors and how it accounted for them,

99:59:59.999 --> 99:59:59.999
but for the purposes of this exercise,

99:59:59.999 --> 99:59:59.999
assume all the information 
you need is included.

99:59:59.999 --> 99:59:59.999
Let’s start by considering the 
cardiovascular effects

99:59:59.999 --> 99:59:59.999
of a certain vitamin, Healthium.

99:59:59.999 --> 99:59:59.999
The study finds that participants taking
Healthium

99:59:59.999 --> 99:59:59.999
had a higher level of healthy cholesterol 
than those taking a placebo.

99:59:59.999 --> 99:59:59.999
Their levels became similar to those of 
people with naturally high levels

99:59:59.999 --> 99:59:59.999
of this kind of cholesterol.

99:59:59.999 --> 99:59:59.999
Previous research has shown that people 
with naturally high levels

99:59:59.999 --> 99:59:59.999
of healthy cholesterol have lower 
rates of heart disease.

99:59:59.999 --> 99:59:59.999
So what makes this headline misleading:

99:59:59.999 --> 99:59:59.999
Healthium reduces risk of heart disease.

99:59:59.999 --> 99:59:59.999
The problem with this headline is that the
research didn’t actually investigate

99:59:59.999 --> 99:59:59.999
whether Healthium reduces heart disease.

99:59:59.999 --> 99:59:59.999
It only measured Healthium’s impact

99:59:59.999 --> 99:59:59.999
on levels of a particular 
kind of cholesterol.

99:59:59.999 --> 99:59:59.999
The fact that people with naturally high 
levels of that cholesterol

99:59:59.999 --> 99:59:59.999
have lower risk of heart attacks

99:59:59.999 --> 99:59:59.999
doesn’t mean that the same will be true

99:59:59.999 --> 99:59:59.999
of people who elevate their cholesterol
levels using Healthium.

99:59:59.999 --> 99:59:59.999
Now that you’ve cracked the 
case of Healthium,

99:59:59.999 --> 99:59:59.999
try your hand at a particularly alluring 
mystery:

99:59:59.999 --> 99:59:59.999
the relationship between eating chocolate
and stress.

99:59:59.999 --> 99:59:59.999
This hypothetical study 
recruits ten students.

99:59:59.999 --> 99:59:59.999
Half begin consuming a 
daily dose of chocolate,

99:59:59.999 --> 99:59:59.999
while half abstain.

99:59:59.999 --> 99:59:59.999
As classmates, they all follow 
the same schedule.

99:59:59.999 --> 99:59:59.999
By the end of the study, the chocolate 
eaters are less stressed

99:59:59.999 --> 99:59:59.999
than their chocolate-free counterparts.

99:59:59.999 --> 99:59:59.999
What’s wrong with this headline:

99:59:59.999 --> 99:59:59.999
Eating chocolate reduces 
stress in students

99:59:59.999 --> 99:59:59.999
It’s a stretch to draw a conclusion about 
students in general from a sample of ten.

99:59:59.999 --> 99:59:59.999
That’s because the fewer participants are
in a random sample,

99:59:59.999 --> 99:59:59.999
the less likely it is that the sample will
closely represent

99:59:59.999 --> 99:59:59.999
the target population as a whole.

99:59:59.999 --> 99:59:59.999
For example, if the broader population of 
students is half male and half female,

99:59:59.999 --> 99:59:59.999
the chance of drawing a sample of 10

99:59:59.999 --> 99:59:59.999
that’s skewed 70% male and 
30% is about 12%.

99:59:59.999 --> 99:59:59.999
In a sample of 100 that would be less than
a .0025% chance,

99:59:59.999 --> 99:59:59.999
and for a sample of 1000,

99:59:59.999 --> 99:59:59.999
the odds are less than 6 x 10^-36.

99:59:59.999 --> 99:59:59.999
Similarly, with fewer participants,

99:59:59.999 --> 99:59:59.999
each individual’s outcome has a larger 
impact on the overall results—

99:59:59.999 --> 99:59:59.999
and can therefore skew big-picture trends.

99:59:59.999 --> 99:59:59.999
Still, there are a lot of good reasons for
scientists run small studies.

99:59:59.999 --> 99:59:59.999
By starting with a small sample,

99:59:59.999 --> 99:59:59.999
they can evaluate whether the results are 
promising enough

99:59:59.999 --> 99:59:59.999
to run a more comprehensive, 
expensive study.

99:59:59.999 --> 99:59:59.999
And some research requires very specific
participants

99:59:59.999 --> 99:59:59.999
that may be impossible to 
recruit in large numbers.

99:59:59.999 --> 99:59:59.999
The key is reproducibility—

99:59:59.999 --> 99:59:59.999
if an article draws a conclusion 
from one small study,

99:59:59.999 --> 99:59:59.999
that conclusion may be suspect—

99:59:59.999 --> 99:59:59.999
but if it’s based on many studies 
that have found similar results,

99:59:59.999 --> 99:59:59.999
it’s more credible.

99:59:59.999 --> 99:59:59.999
We’ve still got one more puzzle.

99:59:59.999 --> 99:59:59.999
In this scenario, a study tests a new drug
for a rare, fatal disease.

99:59:59.999 --> 99:59:59.999
In a sample of 2,000 patients,

99:59:59.999 --> 99:59:59.999
the ones who start taking the drug upon 
diagnosis live longer

99:59:59.999 --> 99:59:59.999
than those who take the placebo.

99:59:59.999 --> 99:59:59.999
This time, the question 
is slightly different.

99:59:59.999 --> 99:59:59.999
What’s one more thing you’d like to know 
before deciding if the headline,

99:59:59.999 --> 99:59:59.999
New drug prolongs lives of patients
with rare disease, is justified?

99:59:59.999 --> 99:59:59.999
Before making this call,

99:59:59.999 --> 99:59:59.999
you’d want to know how much the drug 
prolonged the patients’ lives.

99:59:59.999 --> 99:59:59.999
Sometimes, a study can have results that,

99:59:59.999 --> 99:59:59.999
while scientifically valid, don’t have 
much bearing on real world outcomes.

99:59:59.999 --> 99:59:59.999
For example, one real-life clinical trial 
of a pancreatic cancer drug

99:59:59.999 --> 99:59:59.999
found an increase in life expectancy—
of ten days.

99:59:59.999 --> 99:59:59.999
The next time you see a surprising medical
headline,

99:59:59.999 --> 99:59:59.999
take a look at the science 
it’s reporting on.

99:59:59.999 --> 99:59:59.999
Even when full papers aren’t 
available without a fee,

99:59:59.999 --> 99:59:59.999
you can often find summaries of 
experimental design

99:59:59.999 --> 99:59:59.999
and results in freely available abstracts,

99:59:59.999 --> 99:59:59.999
or even within the text 
of a news articles.

99:59:59.999 --> 99:59:59.999
It’s exciting to see scientific research 
covered in the news,

99:59:59.999 --> 99:59:59.999
and important to understand 
the studies’ findings.