1
99:59:59,999 --> 99:59:59,999
One simple vitamin can reduce your risk of heart disease.

2
99:59:59,999 --> 99:59:59,999
Eating chocolate reduces stress in students.

3
99:59:59,999 --> 99:59:59,999
New drug prolongs lives of patients with rare disease.

4
99:59:59,999 --> 99:59:59,999
Health headlines like these are published
every day,

5
99:59:59,999 --> 99:59:59,999
sometimes making opposite claims 
from each other.

6
99:59:59,999 --> 99:59:59,999
There can be a disconnect between broad,

7
99:59:59,999 --> 99:59:59,999
attention-grabbing headlines 
and the often specific,

8
99:59:59,999 --> 99:59:59,999
incremental results of the medical 
research they cover.

9
99:59:59,999 --> 99:59:59,999
So how can you avoid being 
misled by grabby headlines?

10
99:59:59,999 --> 99:59:59,999
The best way to assess a headline’s 
credibility

11
99:59:59,999 --> 99:59:59,999
is to look at the original 
research it reports on.

12
99:59:59,999 --> 99:59:59,999
We’ve come up with a hypothetical research
scenario

13
99:59:59,999 --> 99:59:59,999
for each of these three headlines.

14
99:59:59,999 --> 99:59:59,999
Keep watching for the explanation
of the first example;

15
99:59:59,999 --> 99:59:59,999
then pause at the headline to 
answer the question.

16
99:59:59,999 --> 99:59:59,999
These are simplified scenarios.

17
99:59:59,999 --> 99:59:59,999
A real study would detail many more 
factors and how it accounted for them,

18
99:59:59,999 --> 99:59:59,999
but for the purposes of this exercise,

19
99:59:59,999 --> 99:59:59,999
assume all the information 
you need is included.

20
99:59:59,999 --> 99:59:59,999
Let’s start by considering the 
cardiovascular effects

21
99:59:59,999 --> 99:59:59,999
of a certain vitamin, Healthium.

22
99:59:59,999 --> 99:59:59,999
The study finds that participants taking
Healthium

23
99:59:59,999 --> 99:59:59,999
had a higher level of healthy cholesterol 
than those taking a placebo.

24
99:59:59,999 --> 99:59:59,999
Their levels became similar to those of 
people with naturally high levels

25
99:59:59,999 --> 99:59:59,999
of this kind of cholesterol.

26
99:59:59,999 --> 99:59:59,999
Previous research has shown that people 
with naturally high levels

27
99:59:59,999 --> 99:59:59,999
of healthy cholesterol have lower 
rates of heart disease.

28
99:59:59,999 --> 99:59:59,999
So what makes this headline misleading:

29
99:59:59,999 --> 99:59:59,999
Healthium reduces risk of heart disease.

30
99:59:59,999 --> 99:59:59,999
The problem with this headline is that the
research didn’t actually investigate

31
99:59:59,999 --> 99:59:59,999
whether Healthium reduces heart disease.

32
99:59:59,999 --> 99:59:59,999
It only measured Healthium’s impact

33
99:59:59,999 --> 99:59:59,999
on levels of a particular 
kind of cholesterol.

34
99:59:59,999 --> 99:59:59,999
The fact that people with naturally high 
levels of that cholesterol

35
99:59:59,999 --> 99:59:59,999
have lower risk of heart attacks

36
99:59:59,999 --> 99:59:59,999
doesn’t mean that the same will be true

37
99:59:59,999 --> 99:59:59,999
of people who elevate their cholesterol
levels using Healthium.

38
99:59:59,999 --> 99:59:59,999
Now that you’ve cracked the 
case of Healthium,

39
99:59:59,999 --> 99:59:59,999
try your hand at a particularly alluring 
mystery:

40
99:59:59,999 --> 99:59:59,999
the relationship between eating chocolate
and stress.

41
99:59:59,999 --> 99:59:59,999
This hypothetical study 
recruits ten students.

42
99:59:59,999 --> 99:59:59,999
Half begin consuming a 
daily dose of chocolate,

43
99:59:59,999 --> 99:59:59,999
while half abstain.

44
99:59:59,999 --> 99:59:59,999
As classmates, they all follow 
the same schedule.

45
99:59:59,999 --> 99:59:59,999
By the end of the study, the chocolate 
eaters are less stressed

46
99:59:59,999 --> 99:59:59,999
than their chocolate-free counterparts.

47
99:59:59,999 --> 99:59:59,999
What’s wrong with this headline:

48
99:59:59,999 --> 99:59:59,999
Eating chocolate reduces 
stress in students

49
99:59:59,999 --> 99:59:59,999
It’s a stretch to draw a conclusion about 
students in general from a sample of ten.

50
99:59:59,999 --> 99:59:59,999
That’s because the fewer participants are
in a random sample,

51
99:59:59,999 --> 99:59:59,999
the less likely it is that the sample will
closely represent

52
99:59:59,999 --> 99:59:59,999
the target population as a whole.

53
99:59:59,999 --> 99:59:59,999
For example, if the broader population of 
students is half male and half female,

54
99:59:59,999 --> 99:59:59,999
the chance of drawing a sample of 10

55
99:59:59,999 --> 99:59:59,999
that’s skewed 70% male and 
30% is about 12%.

56
99:59:59,999 --> 99:59:59,999
In a sample of 100 that would be less than
a .0025% chance,

57
99:59:59,999 --> 99:59:59,999
and for a sample of 1000,

58
99:59:59,999 --> 99:59:59,999
the odds are less than 6 x 10^-36.

59
99:59:59,999 --> 99:59:59,999
Similarly, with fewer participants,

60
99:59:59,999 --> 99:59:59,999
each individual’s outcome has a larger 
impact on the overall results—

61
99:59:59,999 --> 99:59:59,999
and can therefore skew big-picture trends.

62
99:59:59,999 --> 99:59:59,999
Still, there are a lot of good reasons for
scientists run small studies.

63
99:59:59,999 --> 99:59:59,999
By starting with a small sample,

64
99:59:59,999 --> 99:59:59,999
they can evaluate whether the results are 
promising enough

65
99:59:59,999 --> 99:59:59,999
to run a more comprehensive, 
expensive study.

66
99:59:59,999 --> 99:59:59,999
And some research requires very specific
participants

67
99:59:59,999 --> 99:59:59,999
that may be impossible to 
recruit in large numbers.

68
99:59:59,999 --> 99:59:59,999
The key is reproducibility—

69
99:59:59,999 --> 99:59:59,999
if an article draws a conclusion 
from one small study,

70
99:59:59,999 --> 99:59:59,999
that conclusion may be suspect—

71
99:59:59,999 --> 99:59:59,999
but if it’s based on many studies 
that have found similar results,

72
99:59:59,999 --> 99:59:59,999
it’s more credible.

73
99:59:59,999 --> 99:59:59,999
We’ve still got one more puzzle.

74
99:59:59,999 --> 99:59:59,999
In this scenario, a study tests a new drug
for a rare, fatal disease.

75
99:59:59,999 --> 99:59:59,999
In a sample of 2,000 patients,

76
99:59:59,999 --> 99:59:59,999
the ones who start taking the drug upon 
diagnosis live longer

77
99:59:59,999 --> 99:59:59,999
than those who take the placebo.

78
99:59:59,999 --> 99:59:59,999
This time, the question 
is slightly different.

79
99:59:59,999 --> 99:59:59,999
What’s one more thing you’d like to know 
before deciding if the headline,

80
99:59:59,999 --> 99:59:59,999
New drug prolongs lives of patients
with rare disease, is justified?

81
99:59:59,999 --> 99:59:59,999
Before making this call,

82
99:59:59,999 --> 99:59:59,999
you’d want to know how much the drug 
prolonged the patients’ lives.

83
99:59:59,999 --> 99:59:59,999
Sometimes, a study can have results that,

84
99:59:59,999 --> 99:59:59,999
while scientifically valid, don’t have 
much bearing on real world outcomes.

85
99:59:59,999 --> 99:59:59,999
For example, one real-life clinical trial 
of a pancreatic cancer drug

86
99:59:59,999 --> 99:59:59,999
found an increase in life expectancy—
of ten days.

87
99:59:59,999 --> 99:59:59,999
The next time you see a surprising medical
headline,

88
99:59:59,999 --> 99:59:59,999
take a look at the science 
it’s reporting on.

89
99:59:59,999 --> 99:59:59,999
Even when full papers aren’t 
available without a fee,

90
99:59:59,999 --> 99:59:59,999
you can often find summaries of 
experimental design

91
99:59:59,999 --> 99:59:59,999
and results in freely available abstracts,

92
99:59:59,999 --> 99:59:59,999
or even within the text 
of a news articles.

93
99:59:59,999 --> 99:59:59,999
It’s exciting to see scientific research 
covered in the news,

94
99:59:59,999 --> 99:59:59,999
and important to understand 
the studies’ findings.