WEBVTT

00:00:06.912 --> 00:00:11.532
One simple vitamin can reduce 
your risk of heart disease.

00:00:11.532 --> 00:00:15.522
Eating chocolate reduces 
stress in students.

00:00:15.522 --> 00:00:20.222
New drug prolongs lives of 
patients with rare disease.

00:00:20.222 --> 00:00:23.187
Health headlines like these are published
every day,

00:00:23.187 --> 00:00:26.307
sometimes making opposite claims 
from each other.

00:00:26.307 --> 00:00:28.427
There can be a disconnect between broad,

00:00:28.427 --> 00:00:31.537
attention-grabbing headlines 
and the often specific,

00:00:31.537 --> 00:00:34.977
incremental results of the medical 
research they cover.

00:00:34.977 --> 00:00:39.357
So how can you avoid being 
misled by grabby headlines?

00:00:39.357 --> 00:00:42.467
The best way to assess a headline’s 
credibility

00:00:42.467 --> 00:00:45.807
is to look at the original 
research it reports on.

00:00:45.807 --> 00:00:48.217
We’ve come up with a hypothetical research
scenario

00:00:48.217 --> 00:00:50.507
for each of these three headlines.

00:00:50.507 --> 00:00:53.357
Keep watching for the explanation
of the first example;

00:00:53.357 --> 00:00:56.447
then pause at the headline to 
answer the question.

00:00:56.447 --> 00:00:58.517
These are simplified scenarios.

00:00:58.517 --> 00:01:02.827
A real study would detail many more 
factors and how it accounted for them,

00:01:02.827 --> 00:01:04.837
but for the purposes of this exercise,

00:01:04.837 --> 00:01:08.837
assume all the information 
you need is included.

00:01:11.048 --> 00:01:14.098
Let’s start by considering the 
cardiovascular effects

00:01:14.098 --> 00:01:17.068
of a certain vitamin, Healthium.

00:01:17.068 --> 00:01:19.868
The study finds that participants taking
Healthium

00:01:19.868 --> 00:01:24.078
had a higher level of healthy cholesterol 
than those taking a placebo.

00:01:24.078 --> 00:01:28.036
Their levels became similar to those of 
people with naturally high levels

00:01:28.036 --> 00:01:29.936
of this kind of cholesterol.

00:01:29.936 --> 00:01:33.326
Previous research has shown that people 
with naturally high levels

00:01:33.326 --> 00:01:37.486
of healthy cholesterol have lower 
rates of heart disease.

00:01:37.486 --> 00:01:39.991
So what makes this headline misleading:

00:01:39.991 --> 00:01:44.311
Healthium reduces risk of heart disease.

00:01:44.311 --> 00:01:48.791
The problem with this headline is that the
research didn’t actually investigate

00:01:48.791 --> 00:01:51.561
whether Healthium reduces heart disease.

00:01:51.561 --> 00:01:53.651
It only measured Healthium’s impact

00:01:53.651 --> 00:01:56.791
on levels of a particular 
kind of cholesterol.

00:01:56.791 --> 00:01:59.931
The fact that people with naturally high
levels of that cholesterol

00:01:59.931 --> 00:02:01.951
have lower risk of heart attacks

00:02:01.951 --> 00:02:04.391
doesn’t mean that the same 
will be true of people

00:02:04.391 --> 00:02:07.731
who elevate their cholesterol
levels using Healthium.

00:02:07.731 --> 00:02:10.141
Now that you’ve cracked the 
case of Healthium,

00:02:10.141 --> 00:02:13.581
try your hand at a particularly alluring 
mystery:

00:02:13.581 --> 00:02:17.321
the relationship between eating chocolate
and stress.

00:02:17.321 --> 00:02:20.281
This hypothetical study 
recruits ten students.

00:02:20.281 --> 00:02:23.531
Half begin consuming a 
daily dose of chocolate,

00:02:23.531 --> 00:02:25.141
while half abstain.

00:02:25.141 --> 00:02:28.691
As classmates, they all follow 
the same schedule.

00:02:28.691 --> 00:02:32.481
By the end of the study, the chocolate 
eaters are less stressed

00:02:32.481 --> 00:02:35.451
than their chocolate-free counterparts.

00:02:35.451 --> 00:02:37.301
What’s wrong with this headline:

00:02:37.301 --> 00:02:41.371
Eating chocolate reduces 
stress in students

00:02:43.408 --> 00:02:48.638
It’s a stretch to draw a conclusion about 
students in general from a sample of ten.

00:02:48.638 --> 00:02:52.024
That’s because the fewer participants are
in a random sample,

00:02:52.024 --> 00:02:54.884
the less likely it is that the sample will
closely represent

00:02:54.884 --> 00:02:57.934
the target population as a whole.

00:02:57.934 --> 00:03:02.734
For example, if the broader population of 
students is half male and half female,

00:03:02.734 --> 00:03:04.784
the chance of drawing a sample of 10

00:03:04.784 --> 00:03:10.144
that’s skewed 70% male and 
30% is about 12%.

00:03:10.144 --> 00:03:15.693
In a sample of 100 that would be less than
a .0025% chance,

00:03:15.693 --> 00:03:17.525
and for a sample of 1000,

00:03:17.525 --> 00:03:22.795
the odds are less than 6 x 10^-36.

00:03:22.795 --> 00:03:25.137
Similarly, with fewer participants,

00:03:25.137 --> 00:03:29.340
each individual’s outcome has a larger 
impact on the overall results—

00:03:29.340 --> 00:03:32.550
and can therefore skew big-picture trends.

00:03:32.550 --> 00:03:37.490
Still, there are a lot of good reasons for
scientists to run small studies.

00:03:37.490 --> 00:03:39.269
By starting with a small sample,

00:03:39.269 --> 00:03:42.039
they can evaluate whether the results are 
promising enough

00:03:42.039 --> 00:03:45.139
to run a more comprehensive, 
expensive study.

00:03:45.139 --> 00:03:48.609
And some research requires very specific
participants

00:03:48.609 --> 00:03:51.999
that may be impossible to 
recruit in large numbers.

00:03:51.999 --> 00:03:54.329
The key is reproducibility—

00:03:54.329 --> 00:03:57.629
if an article draws a conclusion 
from one small study,

00:03:57.629 --> 00:03:59.649
that conclusion may be suspect—

00:03:59.649 --> 00:04:03.099
but if it’s based on many studies 
that have found similar results,

00:04:03.099 --> 00:04:04.699
it’s more credible.

00:04:04.699 --> 00:04:06.919
We’ve still got one more puzzle.

00:04:06.919 --> 00:04:11.899
In this scenario, a study tests a new drug
for a rare, fatal disease.

00:04:11.899 --> 00:04:14.156
In a sample of 2,000 patients,

00:04:14.156 --> 00:04:17.636
the ones who start taking the drug upon 
diagnosis

00:04:17.636 --> 00:04:21.052
live longer than those who 
take the placebo.

00:04:21.052 --> 00:04:23.642
This time, the question 
is slightly different.

00:04:23.642 --> 00:04:28.012
What’s one more thing you’d like to know 
before deciding if the headline,

00:04:28.012 --> 00:04:33.619
New drug prolongs lives of patients
with rare disease, is justified?

00:04:34.902 --> 00:04:36.472
Before making this call,

00:04:36.472 --> 00:04:40.792
you’d want to know how much the drug 
prolonged the patients’ lives.

00:04:40.792 --> 00:04:43.163
Sometimes, a study can have results that,

00:04:43.163 --> 00:04:48.243
while scientifically valid, don’t have 
much bearing on real world outcomes.

00:04:48.243 --> 00:04:52.941
For example, one real-life clinical trial 
of a pancreatic cancer drug

00:04:52.941 --> 00:04:57.457
found an increase in life expectancy—
of ten days.

00:04:57.457 --> 00:05:00.277
The next time you see a surprising medical
headline,

00:05:00.277 --> 00:05:04.027
take a look at the science 
it’s reporting on.

00:05:04.027 --> 00:05:06.957
Even when full papers aren’t 
available without a fee,

00:05:06.957 --> 00:05:09.987
you can often find summaries of 
experimental design

00:05:09.987 --> 00:05:13.177
and results in freely available abstracts,

00:05:13.177 --> 00:05:16.361
or even within the text 
of a news article.

00:05:16.361 --> 00:05:19.671
It’s exciting to see scientific research 
covered in the news,

00:05:19.671 --> 00:05:23.951
and important to understand 
the studies’ findings.