1
00:00:00,246 --> 00:00:04,366
>> I'm here at the [inaudible] dying of heat
exhaustion to talk to you about these guys.

2
00:00:05,126 --> 00:00:08,896
Specifically, how they relate to one
of our biggest problems in science.

3
00:00:08,896 --> 00:00:10,406
Let's start from the beginning.

4
00:00:10,706 --> 00:00:13,876
There is a replication crisis
among scientists and it's serious.

5
00:00:14,566 --> 00:00:17,646
Several scientific fields have
done major replication attempts

6
00:00:17,646 --> 00:00:20,616
and they just aren't getting the
same results the second time around.

7
00:00:20,986 --> 00:00:24,256
And this isn't a case of an
odd finding here or there, no.

8
00:00:24,496 --> 00:00:29,626
We're having to question foundational studies
in the literature of psychology, biomedicine.

9
00:00:29,766 --> 00:00:32,496
And while the harder scientists
might be smugly chuckling right now

10
00:00:32,496 --> 00:00:35,186
about their epistemic superiority, many wonder

11
00:00:35,216 --> 00:00:38,736
if they're not a big replication
attempt away from the same situation.

12
00:00:38,966 --> 00:00:43,836
In short, the replication crisis is a
serious problem and not just for scientists.

13
00:00:44,006 --> 00:00:47,076
Remember, what happens in
science becomes public policy.

14
00:00:47,176 --> 00:00:49,446
What happens in science becomes
your drug treatment.

15
00:00:49,626 --> 00:00:53,876
What happens in science turns into pop
science which turns into your friend insisting

16
00:00:53,876 --> 00:00:58,436
at a dinner party that power-posing is the
secret to becoming dominant and assertive.

17
00:00:58,516 --> 00:00:59,446
>> There is nothing there.

18
00:00:59,586 --> 00:01:01,786
The books have nothing to say.

19
00:01:02,516 --> 00:01:05,426
[ Music ]

20
00:01:05,926 --> 00:01:08,896
>> We all have a vested interest
in science getting it right

21
00:01:08,966 --> 00:01:11,316
or at least correcting itself when it's wrong.

22
00:01:11,636 --> 00:01:15,766
But what's so terrifying about the replication
crisis isn't that we got a few things wrong,

23
00:01:15,936 --> 00:01:18,576
it's that we didn't notice
they were wrong for so long.

24
00:01:18,576 --> 00:01:21,966
The explanations for the
replication crisis are varied.

25
00:01:22,246 --> 00:01:26,086
A few simply deny there is a problem;
others propose that our standards

26
00:01:26,086 --> 00:01:28,106
for statistical significance are too weak.

27
00:01:28,436 --> 00:01:32,846
P-values, they would say, breeds too many false
positives and have helped create the crisis.

28
00:01:33,236 --> 00:01:35,516
Some say we don't incentivize
replication enough.

29
00:01:35,656 --> 00:01:40,066
Some say the gate-keeping institutions are
to blame and the culprit list goes on and on.

30
00:01:40,066 --> 00:01:43,076
And I think most of these have some
truth to them, but I want to talk

31
00:01:43,076 --> 00:01:45,576
about a much simpler theory
for the replication crisis,

32
00:01:45,606 --> 00:01:50,736
a theory that begins with
us looking at giraffes.

33
00:01:50,736 --> 00:01:52,906
This video is sponsored by Squarespace.

34
00:01:53,006 --> 00:01:58,406
Not this exact giraffe of course but the
process that led us to this exact giraffe.

35
00:01:58,596 --> 00:02:02,696
Natural selection is the process
where organisms better adapted

36
00:02:02,696 --> 00:02:05,446
to an environment tend to survive and reproduce.

37
00:02:05,526 --> 00:02:07,706
And giraffes are an archetype
of natural selection.

38
00:02:07,766 --> 00:02:10,376
The ones with longer necks were
better able to eat tree leaves

39
00:02:10,376 --> 00:02:13,086
and reproduce and over time you get this.

40
00:02:13,646 --> 00:02:16,296
But the scientist Dr. Paul
Smaldino [assumed spelling] argues

41
00:02:16,336 --> 00:02:20,636
that this same process affects
more than just animals.

42
00:02:20,636 --> 00:02:25,506
>> So if you can imagine, we're used to
thinking about evolution in terms of acting

43
00:02:25,806 --> 00:02:31,366
on [inaudible] and on biological species that
die and reproduce but culture also evolves.

44
00:02:31,366 --> 00:02:35,976
And, you know, Darwin laid out --
you don't need genes for evolution.

45
00:02:36,086 --> 00:02:38,086
There are just -- you just need three things.

46
00:02:38,086 --> 00:02:41,676
Right. You need variation in a population.

47
00:02:41,676 --> 00:02:46,716
You need the things that vary to be heritable
to be able to be passed down from one individual

48
00:02:46,716 --> 00:02:51,556
to another and you need selection so
that variation matters in the extent

49
00:02:51,556 --> 00:02:53,316
to which those traits are passed down.

50
00:02:53,316 --> 00:02:58,636
And anything that has those properties
is subject to natural selection.

51
00:02:58,636 --> 00:03:00,306
>> Anything including science.

52
00:03:00,486 --> 00:03:01,836
>> So scientists vary in their methods.

53
00:03:01,836 --> 00:03:06,886
There is heritability in which, you
know, different methods get passed

54
00:03:06,886 --> 00:03:13,576
down so students have advisors who they
learn from, professors who have influence.

55
00:03:13,576 --> 00:03:17,086
They spread their methods through
their papers, through their seminars,

56
00:03:17,086 --> 00:03:20,166
through their lectures and their selection.

57
00:03:20,166 --> 00:03:27,836
There is a lot more people who get -- who is in
the sciences who want jobs than can have them

58
00:03:27,836 --> 00:03:30,276
and that -- so there is a bottleneck.

59
00:03:30,276 --> 00:03:34,636
And selection through the
bottleneck is non-random and this --

60
00:03:34,636 --> 00:03:38,976
you know, so the question is then what
are the traits that we're selecting for?

61
00:03:39,516 --> 00:03:41,546
[ Music ]

62
00:03:42,046 --> 00:03:47,696
>> In theory, we're selecting for truth but
it's hard to know what's true so we've tended

63
00:03:47,696 --> 00:03:50,086
to use publications in journals as a proxy.

64
00:03:50,086 --> 00:03:55,676
So what we actually select for in practice
are as many publications as possible

65
00:03:55,676 --> 00:04:00,046
in the best journals possible
which Smaldino admits sounds good.

66
00:04:00,716 --> 00:04:02,956
>> You know, we want to hire
scientists who are productive,

67
00:04:02,956 --> 00:04:05,546
who do a lot of -- you know
actually produce work.

68
00:04:05,546 --> 00:04:11,206
We want to hire scientists whose work is
impactful, that gets cited a lot, gets --

69
00:04:11,206 --> 00:04:16,876
you know is important enough to be in
high-impact journals and get press coverage.

70
00:04:17,186 --> 00:04:25,196
The issue is that if we use the measure
of how many papers, the impact of those --

71
00:04:25,196 --> 00:04:29,006
of the journal they're published
in, how many grants, right,

72
00:04:29,056 --> 00:04:33,646
these are proxies and proxies can be gamed.

73
00:04:33,716 --> 00:04:36,946
>> Proxies cannot only be
gamed, they will be gamed.

74
00:04:37,026 --> 00:04:39,886
This is something called
Goodhart's law in economics.

75
00:04:40,366 --> 00:04:41,816
I'm going to demonstrate how.

76
00:04:42,616 --> 00:04:45,886
Let's say you have a gold mine, and
we'll put it on Science Mountain.

77
00:04:45,886 --> 00:04:49,116
Now normally the miners just look
for gold and sell it to merchants

78
00:04:49,506 --> 00:04:52,316
but unfortunately there are
limited mines on Science Mountain.

79
00:04:52,316 --> 00:04:54,156
So to make sure you have
the most productive miners,

80
00:04:54,326 --> 00:04:56,546
you think it's a good idea
to measure productivity.

81
00:04:56,546 --> 00:05:00,646
So you tell miners, the ones who sell
the most gold every year keep their jobs

82
00:05:00,646 --> 00:05:01,446
and get promoted.

83
00:05:01,776 --> 00:05:02,186
Fair enough.

84
00:05:02,876 --> 00:05:04,976
But there is a small problem
on Science Mountain.

85
00:05:05,086 --> 00:05:08,486
Gold is rare but there is also
fools gold which is less rare.

86
00:05:08,726 --> 00:05:10,206
Fools gold looks like gold.

87
00:05:10,206 --> 00:05:12,096
It feels like gold but it's not gold.

88
00:05:12,166 --> 00:05:14,646
And as the owner of the mine,
you don't want fools gold.

89
00:05:14,646 --> 00:05:16,406
It ruins your reputation long-term.

90
00:05:16,816 --> 00:05:20,976
Now we'd hope that merchants would be
really good at spotting fools gold.

91
00:05:21,256 --> 00:05:24,046
They wouldn't buy it, so to
speak, so that we keep everything

92
00:05:24,046 --> 00:05:26,716
in check but we find that's not the case.

93
00:05:26,826 --> 00:05:28,346
Science Mountain has fools gold.

94
00:05:28,386 --> 00:05:32,056
It's is really hard to spot, and the merchants
aren't that good at telling the difference.

95
00:05:32,366 --> 00:05:36,096
The only people who are really experts
are the miners who specialize in that mine

96
00:05:36,526 --> 00:05:40,366
but now you've given them every incentive
to work fast and ask questions later.

97
00:05:40,456 --> 00:05:44,176
It's not hard to see that given that
situation, in a very short amount of time,

98
00:05:44,176 --> 00:05:48,996
you'd flood the market with fools gold
because whatever could be found and polished

99
00:05:48,996 --> 00:05:51,156
up and sent off as gold would be.

100
00:05:51,156 --> 00:05:55,626
Meanwhile, your scrupulous miners, the
ones who took their time and made sure only

101
00:05:55,626 --> 00:05:59,076
to sell real gold would be fired
for not being productive enough.

102
00:05:59,996 --> 00:06:01,886
This is good Goodhart's law at work.

103
00:06:02,276 --> 00:06:05,386
You picked a measure of productivity,
made it a target,

104
00:06:05,386 --> 00:06:09,226
and inadvertently made all
your miners less productive.

105
00:06:09,396 --> 00:06:12,826
Your miners are busy filling quotas
to try to keep their job instead

106
00:06:12,826 --> 00:06:15,456
of making sure what they're
mining is really gold.

107
00:06:15,946 --> 00:06:18,116
Now this, of course, is more
than a simple analogy.

108
00:06:18,156 --> 00:06:20,116
This is the situation with real science.

109
00:06:20,786 --> 00:06:23,656
Journals are not good at
catching false positives.

110
00:06:23,926 --> 00:06:28,186
So when you make publication numbers
and citations the measure of a scientist

111
00:06:28,516 --> 00:06:32,786
and what gets published are sensational
results, you might think you're selecting

112
00:06:32,786 --> 00:06:35,866
for more true, exciting science but you're not.

113
00:06:36,086 --> 00:06:38,896
You're just going to end up with a lot
of exciting science that appears true

114
00:06:39,026 --> 00:06:41,366
but may, in fact, be built on nothing.

115
00:06:41,636 --> 00:06:44,866
And it doesn't help that although
we pay lip service to replication,

116
00:06:45,156 --> 00:06:48,236
we don't give scientists much incentive
to replicate old studies which means

117
00:06:48,236 --> 00:06:51,106
that once an untruth gets into
our scientific literature,

118
00:06:51,396 --> 00:06:53,376
it's really hard for us to dig it back out.

119
00:06:53,846 --> 00:06:57,056
>> Because of the publish or perish mentality,

120
00:06:57,436 --> 00:07:02,826
we are currently facing an impossible
amount of scientific articles.

121
00:07:03,196 --> 00:07:09,726
The well-respected, "The Lancet" researched
that a while ago and they found that 85%

122
00:07:09,776 --> 00:07:16,586
of the published biomedical
research is rubbish, nonsense.

123
00:07:16,786 --> 00:07:22,336
If you throw in enough data and you give it a
good scientific stir, you'd be a complete idiot

124
00:07:22,336 --> 00:07:26,976
if you don't find any correlation
between something and something else

125
00:07:27,456 --> 00:07:36,686
and then your published and your H-index goes
up and that, my dear scientist of the future,

126
00:07:37,736 --> 00:07:43,776
is a good thing; not for society,
not for the world but for you.

127
00:07:44,516 --> 00:07:47,546
[ Music ]

128
00:07:48,046 --> 00:07:52,506
>> What is good for a scientist's
career isn't good for science.

129
00:07:52,506 --> 00:07:55,136
That is at the heart of Smaldino's critique.

130
00:07:55,136 --> 00:07:59,256
But he goes a step further and says
look, this isn't an individual problem.

131
00:07:59,256 --> 00:08:02,946
This isn't a matter of some bad
scientists ruining it for the rest.

132
00:08:02,946 --> 00:08:06,806
Natural selection doesn't require
conscious strategizing and here is

133
00:08:06,806 --> 00:08:09,186
where our giraffe example is once again useful.

134
00:08:09,186 --> 00:08:12,546
No individual giraffe conspired
to get a longer neck.

135
00:08:12,546 --> 00:08:17,416
In the same way, no scientist even
needs to be aware of the system in order

136
00:08:17,466 --> 00:08:19,836
for the replication crisis to happen.

137
00:08:19,836 --> 00:08:22,746
This is a problem that Smaldino
things a lot of people miss.

138
00:08:22,746 --> 00:08:30,256
>> If the incentives are still there, then
over time everyone can still have, you know,

139
00:08:30,256 --> 00:08:34,476
perfectly good intentions and
say I'm going to be my best

140
00:08:34,476 --> 00:08:39,716
but the people whose best involves
doing, let's say, less rigorous work

141
00:08:39,716 --> 00:08:46,126
and getting more papers out, doing less
rigorous work and over-hyping the results

142
00:08:46,126 --> 00:08:52,916
or interpreting it in ways that, let's
say, have less integrity but, you know,

143
00:08:52,916 --> 00:08:55,806
get the headline, they're going to get rewarded.

144
00:08:55,806 --> 00:08:57,796
They're going to get the better jobs.

145
00:08:57,796 --> 00:08:59,256
They're going to get more grants.

146
00:08:59,256 --> 00:09:01,026
They're going to attract more students.

147
00:09:01,026 --> 00:09:06,306
Their students are going to copy their methods
and those methods are going to propagate

148
00:09:06,306 --> 00:09:09,616
that lead to worse results,
more false positives,

149
00:09:09,616 --> 00:09:11,946
new false discoveries, less reproducibility.

150
00:09:11,946 --> 00:09:14,546
>> And the problems don't stop there.

151
00:09:14,546 --> 00:09:17,646
It's not just about bad research
being able to thrive.

152
00:09:17,916 --> 00:09:22,426
Having a constant supply of so-called
groundbreaking studies also devalues what we

153
00:09:22,506 --> 00:09:24,656
think of as true in science.

154
00:09:24,656 --> 00:09:29,036
Consider that in the last 40 years, words
like innovative, groundbreaking and novel

155
00:09:29,036 --> 00:09:33,706
in PubMed journal abstracts
have increased by 2,500%.

156
00:09:33,706 --> 00:09:37,166
Now one would hope that scientists
have gotten that much more innovative

157
00:09:37,166 --> 00:09:39,766
in 40 years but that seems unlikely.

158
00:09:39,766 --> 00:09:43,846
Instead, it seems indicative of an
ever-increasing pressure on scientists

159
00:09:44,156 --> 00:09:48,706
to be pumping out more and more papers every
year to be able to compete with their colleagues

160
00:09:48,706 --> 00:09:51,006
so they don't lose their
spot on Science Mountain.

161
00:09:51,246 --> 00:09:55,316
And in that environment, the
scientists who choose to do deep, slow,

162
00:09:55,316 --> 00:09:59,216
rigorous work fall farther and farther behind.

163
00:09:59,216 --> 00:10:02,766
You might have heard of Peter Higgs.

164
00:10:03,046 --> 00:10:06,676
He won the Nobel Prize for physics in
2013 for predicting the Higgs bosons.

165
00:10:06,676 --> 00:10:09,906
But what you might not appreciate
is that-that piece

166
00:10:09,906 --> 00:10:12,746
of great science took Peter
Higgs a very long time.

167
00:10:12,836 --> 00:10:17,686
It took him five years of publishing nothing,
just working on this one problem in order

168
00:10:17,686 --> 00:10:20,806
to publish a paper that he would
eventually get the Nobel Prize for.

169
00:10:21,106 --> 00:10:23,956
In an interview, Peter Higgs said
he couldn't do that work today.

170
00:10:24,186 --> 00:10:25,126
He'd never be hired.

171
00:10:25,216 --> 00:10:27,936
He says he isn't productive
enough by today's standards.

172
00:10:28,146 --> 00:10:34,406
>> And to do deep work also
requires some removal.

173
00:10:34,536 --> 00:10:38,516
Right. If you want to do really deep,
interesting, unique work, right, who --

174
00:10:38,516 --> 00:10:44,876
if you -- and this is true of any field and not
just science or art and, you know, literature.

175
00:10:44,876 --> 00:10:50,596
Anything that involves creation, right, the
people who are going to be doing the things

176
00:10:50,596 --> 00:10:55,286
that are the most unique and potentially
groundbreaking are probably not the people

177
00:10:55,286 --> 00:10:58,676
who are always moving and hustling
and, you know, in the limelight.

178
00:10:58,726 --> 00:11:06,846
It's going to be people who at least sometimes
take time and go away for a while and work

179
00:11:06,846 --> 00:11:08,936
in a concentrated way on what they're doing.

180
00:11:09,206 --> 00:11:14,436
And if you never have the time and the space
to do that, if the people who need the time

181
00:11:14,436 --> 00:11:19,906
and the space to do that are never given the
opportunity to do that, you know what does

182
00:11:19,906 --> 00:11:22,996
that mean for the landscape of research?

183
00:11:22,996 --> 00:11:26,726
You know the kinds of research
that we see that gets done?

184
00:11:27,876 --> 00:11:33,866
It's not obvious how to solve this problem, but
I think it's an important thing to consider.

185
00:11:36,096 --> 00:11:36,676
>> Wait. Wait.

186
00:11:36,676 --> 00:11:37,076
Wait. Wait.

187
00:11:37,076 --> 00:11:37,636
Wait. Wait.

188
00:11:37,636 --> 00:11:39,396
We can't end it like that.

189
00:11:39,396 --> 00:11:41,226
We haven't even talked about solutions.

190
00:11:41,226 --> 00:11:42,396
And we are about to fix that.

191
00:11:42,396 --> 00:11:46,106
We will talk about solutions to these
very big, very real problems in science

192
00:11:46,106 --> 00:11:48,956
but first we're going to talk
about a much simpler solution

193
00:11:48,956 --> 00:11:50,736
to a big problem which is the website.

194
00:11:51,016 --> 00:11:53,736
So thank you to our sponsor today, Squarespace.

195
00:11:54,056 --> 00:11:56,396
Bad joke but great website builder.

196
00:11:56,606 --> 00:12:01,316
Squarespace is your all-in-one spot for
any kind of website you want to build.

197
00:12:01,636 --> 00:12:06,136
Now I know I talked in the last video about
how it's super customizable but really easy

198
00:12:06,136 --> 00:12:08,876
to use and surprisingly a deep product.

199
00:12:08,876 --> 00:12:11,846
But what I want to focus on
today is how fast it loads.

200
00:12:11,846 --> 00:12:15,436
Now they use progressive image loading which
means that when someone goes on your site --

201
00:12:15,746 --> 00:12:17,476
normally if you build a website yourself,

202
00:12:17,476 --> 00:12:19,236
your images probably all try
to load at the same time.

203
00:12:19,526 --> 00:12:20,316
It's slow.

204
00:12:20,406 --> 00:12:24,756
Progressive image loading loads the top
images first so your website is very snappy.

205
00:12:24,796 --> 00:12:26,936
We all know that snappy websites are the king.

206
00:12:26,936 --> 00:12:31,226
You don't want to get that back button we've all
click before because a website's not loading.

207
00:12:31,226 --> 00:12:32,466
You don't want that to happen on your website.

208
00:12:32,466 --> 00:12:36,336
So Squarespace, it solves all these
things for you so you don't have

209
00:12:36,416 --> 00:12:37,766
to spend time worrying about them.

210
00:12:38,136 --> 00:12:40,976
So if you want to go check it
out, it's a 14 day free trial.

211
00:12:41,046 --> 00:12:44,186
You can build the website, try all the
little features, and when you're sure

212
00:12:44,186 --> 00:12:47,736
that it does everything that you want
it to do, you can go to check out

213
00:12:47,736 --> 00:12:51,866
and use promo code coffee break for
10% off your first domain or website.

214
00:12:51,866 --> 00:12:56,146
Once again, that's promo code coffee break
for 10% off your first website or domain.

215
00:12:56,526 --> 00:12:58,516
Thank you to Squarespace
for sponsoring this video.

216
00:12:58,796 --> 00:13:00,986
Now let's talk about much
more complicated solutions,

217
00:13:00,986 --> 00:13:02,816
how do we fix these problems in science?

218
00:13:03,086 --> 00:13:07,016
So I put two full interviews together with
two really smart scientists all about this.

219
00:13:07,266 --> 00:13:08,356
One is Dr. Smaldino.

220
00:13:08,356 --> 00:13:10,626
You've seen him before and
number two is Dr. [inaudible].

221
00:13:11,346 --> 00:13:12,086
Super great.

222
00:13:12,086 --> 00:13:13,976
You're going to get a lot
out of that conversation,

223
00:13:13,976 --> 00:13:19,626
but if you just want a little taste, here's
a little teaser for those two interviews.

224
00:13:19,626 --> 00:13:20,446
Thank you for watching.

225
00:13:20,446 --> 00:13:23,446
Thank you for supporting the channel,
and I will see you guys next time.

226
00:13:23,446 --> 00:13:24,336
>> So let me be clear.

227
00:13:24,336 --> 00:13:26,696
Like science is this great process.

228
00:13:26,696 --> 00:13:29,816
It is like the best way to discover truth.

229
00:13:29,816 --> 00:13:34,306
>> We have the notion that replication
is boring is demonstrably false.

230
00:13:34,306 --> 00:13:36,636
>> There is, you know, this nice
game theory modeling showing

231
00:13:36,636 --> 00:13:41,676
that a lottery would actually be a
really good use of the scientist's time.

232
00:13:41,986 --> 00:13:44,296
>> The best intervention that we have

233
00:13:44,296 --> 00:13:47,976
so far that's realigning the
incentives is Registered Reports.

234
00:13:48,516 --> 00:13:53,220
[ Music ]