1
00:00:06,912 --> 00:00:11,532
One simple vitamin can reduce 
your risk of heart disease.

2
00:00:11,532 --> 00:00:15,522
Eating chocolate reduces 
stress in students.

3
00:00:15,522 --> 00:00:20,222
New drug prolongs lives of 
patients with rare disease.

4
00:00:20,222 --> 00:00:23,187
Health headlines like these 
are published every day,

5
00:00:23,187 --> 00:00:26,307
sometimes making opposite claims 
from each other.

6
00:00:26,307 --> 00:00:28,427
There can be a disconnect between broad,

7
00:00:28,427 --> 00:00:31,537
attention-grabbing headlines 
and the often specific,

8
00:00:31,537 --> 00:00:34,977
incremental results of the medical 
research they cover.

9
00:00:34,977 --> 00:00:39,357
So how can you avoid being 
misled by grabby headlines?

10
00:00:39,357 --> 00:00:42,467
The best way to assess a headline’s 
credibility

11
00:00:42,467 --> 00:00:45,807
is to look at the original 
research it reports on.

12
00:00:45,807 --> 00:00:48,217
We’ve come up with a hypothetical research
scenario

13
00:00:48,217 --> 00:00:50,507
for each of these three headlines.

14
00:00:50,507 --> 00:00:53,357
Keep watching for the explanation
of the first example;

15
00:00:53,357 --> 00:00:56,447
then pause at the headline to 
answer the question.

16
00:00:56,447 --> 00:00:58,517
These are simplified scenarios.

17
00:00:58,517 --> 00:01:02,827
A real study would detail many more 
factors and how it accounted for them,

18
00:01:02,827 --> 00:01:04,837
but for the purposes of this exercise,

19
00:01:04,837 --> 00:01:08,837
assume all the information 
you need is included.

20
00:01:11,048 --> 00:01:14,098
Let’s start by considering the 
cardiovascular effects

21
00:01:14,098 --> 00:01:17,068
of a certain vitamin, Healthium.

22
00:01:17,068 --> 00:01:19,868
The study finds that participants taking
Healthium

23
00:01:19,868 --> 00:01:24,078
had a higher level of healthy cholesterol 
than those taking a placebo.

24
00:01:24,078 --> 00:01:28,036
Their levels became similar to those of 
people with naturally high levels

25
00:01:28,036 --> 00:01:29,936
of this kind of cholesterol.

26
00:01:29,936 --> 00:01:33,326
Previous research has shown that people 
with naturally high levels

27
00:01:33,326 --> 00:01:37,486
of healthy cholesterol have lower 
rates of heart disease.

28
00:01:37,486 --> 00:01:39,991
So what makes this headline misleading:

29
00:01:39,991 --> 00:01:44,311
"Healthium reduces risk of heart disease."

30
00:01:44,311 --> 00:01:48,791
The problem with this headline is that the
research didn’t actually investigate

31
00:01:48,791 --> 00:01:51,561
whether Healthium reduces heart disease.

32
00:01:51,561 --> 00:01:53,651
It only measured Healthium’s impact

33
00:01:53,651 --> 00:01:56,791
on levels of a particular 
kind of cholesterol.

34
00:01:56,791 --> 00:01:59,931
The fact that people with naturally high
levels of that cholesterol

35
00:01:59,931 --> 00:02:01,951
have lower risk of heart attacks

36
00:02:01,951 --> 00:02:04,391
doesn’t mean that the same 
will be true of people

37
00:02:04,391 --> 00:02:07,731
who elevate their cholesterol
levels using Healthium.

38
00:02:07,731 --> 00:02:10,141
Now that you’ve cracked the 
case of Healthium,

39
00:02:10,141 --> 00:02:13,581
try your hand at a particularly alluring 
mystery:

40
00:02:13,581 --> 00:02:17,321
the relationship between eating chocolate
and stress.

41
00:02:17,321 --> 00:02:20,281
This hypothetical study 
recruits ten students.

42
00:02:20,281 --> 00:02:23,531
Half begin consuming a 
daily dose of chocolate,

43
00:02:23,531 --> 00:02:25,141
while half abstain.

44
00:02:25,141 --> 00:02:28,691
As classmates, they all follow 
the same schedule.

45
00:02:28,691 --> 00:02:32,481
By the end of the study, the chocolate 
eaters are less stressed

46
00:02:32,481 --> 00:02:35,451
than their chocolate-free counterparts.

47
00:02:35,451 --> 00:02:37,301
What’s wrong with this headline:

48
00:02:37,301 --> 00:02:41,371
"Eating chocolate reduces 
stress in students"

49
00:02:43,408 --> 00:02:48,638
It’s a stretch to draw a conclusion about 
students in general from a sample of ten.

50
00:02:48,638 --> 00:02:52,024
That’s because the fewer participants are
in a random sample,

51
00:02:52,024 --> 00:02:54,884
the less likely it is that the sample will
closely represent

52
00:02:54,884 --> 00:02:57,934
the target population as a whole.

53
00:02:57,934 --> 00:03:02,734
For example, if the broader population of 
students is half male and half female,

54
00:03:02,734 --> 00:03:04,784
the chance of drawing a sample of 10

55
00:03:04,784 --> 00:03:10,144
that’s skewed 70% male and 
30% is about 12%.

56
00:03:10,144 --> 00:03:15,693
In a sample of 100 that would be less than
a .0025% chance,

57
00:03:15,693 --> 00:03:17,525
and for a sample of 1000,

58
00:03:17,525 --> 00:03:22,795
the odds are less than 6 x 10^-36.

59
00:03:22,795 --> 00:03:25,137
Similarly, with fewer participants,

60
00:03:25,137 --> 00:03:29,340
each individual’s outcome has a larger 
impact on the overall results—

61
00:03:29,340 --> 00:03:32,550
and can therefore skew big-picture trends.

62
00:03:32,550 --> 00:03:37,490
Still, there are a lot of good reasons for
scientists to run small studies.

63
00:03:37,490 --> 00:03:39,269
By starting with a small sample,

64
00:03:39,269 --> 00:03:42,039
they can evaluate whether the results are 
promising enough

65
00:03:42,039 --> 00:03:45,139
to run a more comprehensive, 
expensive study.

66
00:03:45,139 --> 00:03:48,609
And some research requires very specific
participants

67
00:03:48,609 --> 00:03:51,999
that may be impossible to 
recruit in large numbers.

68
00:03:51,999 --> 00:03:54,329
The key is reproducibility—

69
00:03:54,329 --> 00:03:57,629
if an article draws a conclusion 
from one small study,

70
00:03:57,629 --> 00:03:59,649
that conclusion may be suspect—

71
00:03:59,649 --> 00:04:03,099
but if it’s based on many studies 
that have found similar results,

72
00:04:03,099 --> 00:04:04,699
it’s more credible.

73
00:04:04,699 --> 00:04:06,919
We’ve still got one more puzzle.

74
00:04:06,919 --> 00:04:11,899
In this scenario, a study tests a new drug
for a rare, fatal disease.

75
00:04:11,899 --> 00:04:14,156
In a sample of 2,000 patients,

76
00:04:14,156 --> 00:04:17,636
the ones who start taking the drug upon 
diagnosis

77
00:04:17,636 --> 00:04:21,052
live longer than those who 
take the placebo.

78
00:04:21,052 --> 00:04:23,642
This time, the question 
is slightly different.

79
00:04:23,642 --> 00:04:28,012
What’s one more thing you’d like to know 
before deciding if the headline,

80
00:04:28,012 --> 00:04:33,619
"New drug prolongs lives of patients
with rare disease", is justified?

81
00:04:34,902 --> 00:04:36,472
Before making this call,

82
00:04:36,472 --> 00:04:40,792
you’d want to know how much the drug 
prolonged the patients’ lives.

83
00:04:40,792 --> 00:04:43,163
Sometimes, a study can have results that,

84
00:04:43,163 --> 00:04:48,243
while scientifically valid, don’t have 
much bearing on real world outcomes.

85
00:04:48,243 --> 00:04:52,941
For example, one real-life clinical trial 
of a pancreatic cancer drug

86
00:04:52,941 --> 00:04:57,457
found an increase in life expectancy—
of ten days.

87
00:04:57,457 --> 00:05:00,277
The next time you see a surprising medical
headline,

88
00:05:00,277 --> 00:05:04,027
take a look at the science 
it’s reporting on.

89
00:05:04,027 --> 00:05:06,957
Even when full papers aren’t 
available without a fee,

90
00:05:06,957 --> 00:05:09,987
you can often find summaries of 
experimental design

91
00:05:09,987 --> 00:05:13,177
and results in freely available abstracts,

92
00:05:13,177 --> 00:05:16,361
or even within the text 
of a news article.

93
00:05:16,361 --> 00:05:19,671
It’s exciting to see scientific research 
covered in the news,

94
00:05:19,671 --> 00:05:23,401
and important to understand 
the studies’ findings.