1
00:00:00,000 --> 00:00:18,751
36c3 Intro Music
2
00:00:19,571 --> 00:00:23,300
Herald: ...now with the talk "The useful-
harmless spectrum". As I said,
3
00:00:23,300 --> 00:00:26,625
he needs no introduction: Fefe.
4
00:00:26,625 --> 00:00:30,482
Applause
5
00:00:30,482 --> 00:00:32,472
Tapping on the microphone
6
00:00:37,082 --> 00:00:40,110
Fefe: Good morning, I am happy that there
are so many people here.
7
00:00:40,110 --> 00:00:44,340
Thankfully this is not Hall 1. That would
be bad, with so many people.
8
00:00:44,340 --> 00:00:46,770
I have to manage your expectations
before I start,
9
00:00:46,770 --> 00:00:50,820
I actually submitted a different talk
last year about TCB-minimization,
10
00:00:50,820 --> 00:00:54,240
which would have been a bit technical,
about what you can do
11
00:00:54,240 --> 00:00:59,670
as a programmer. It was not accepted,
I don't know why - schedule was full.
12
00:00:59,670 --> 00:01:02,520
I submitted it again this year,
but I didn't want it to look
13
00:01:02,520 --> 00:01:05,520
like I want to bother them, so I
submitted another talk.
14
00:01:05,520 --> 00:01:10,800
...of course they accepted that one.
Which meant, I had to
15
00:01:10,800 --> 00:01:12,810
quickly prepare it now.
Audience laughs
16
00:01:13,320 --> 00:01:19,290
Well, the problem is, this is more of a
thought process than a structured
17
00:01:19,290 --> 00:01:23,490
presentation. I hope that it'll be
helpful none the less. But it's
18
00:01:23,490 --> 00:01:27,720
not as structured as my usual talks.
I will just start. So, there are multiple
19
00:01:27,720 --> 00:01:32,310
approaches, that basically result in
the same result, and I will
20
00:01:32,310 --> 00:01:36,540
just let you listen. Relatively early in
my career, I decided the following:
21
00:01:36,540 --> 00:01:40,980
I will never write software on which
people's lives may depend,
22
00:01:40,980 --> 00:01:45,780
like medical devices, nuclear reactors;
that was my idea.
23
00:01:45,780 --> 00:01:51,120
Of course not military either. And then
I met somebody that writes code for
24
00:01:51,120 --> 00:01:54,450
nuclear reactors. And it was the kind of
guy that says "That's super easy"
25
00:01:54,450 --> 00:02:00,300
So when those that know their
limits don't do it, then
26
00:02:00,300 --> 00:02:03,660
the other people will.
Audience laughs
27
00:02:04,630 --> 00:02:08,980
I don't want to generalize though.
I also met another guy that
28
00:02:08,980 --> 00:02:12,160
was not like this, but I mean,
this type of person exist.
29
00:02:12,160 --> 00:02:18,220
I believe that the problem here
is that you learn programming
30
00:02:18,220 --> 00:02:23,350
exploratively: It's not like a set path,
on which you walk, but rather you
31
00:02:23,350 --> 00:02:28,690
are just walking around and finding
your limits. But by definition this also
32
00:02:28,690 --> 00:02:33,160
means, that you don't know your limits
yet, because you are looking for them.
33
00:02:33,160 --> 00:02:38,260
This also means that you are always
working at your limit though. When people
34
00:02:38,260 --> 00:02:41,200
write software, then they go
just as far as they believe they
35
00:02:41,200 --> 00:02:47,320
can just barely go. In turn, this also
means that the technology that
36
00:02:47,320 --> 00:02:50,530
is being rolled out out there is mainly
not tried and tested
37
00:02:50,530 --> 00:02:55,240
or well understood, but rather it is the
technology, that the programmer
38
00:02:55,240 --> 00:03:01,450
just barely still understood. This is a
bit of a problem, which is further
39
00:03:01,450 --> 00:03:04,660
amplified by today's modularization and
dependency wave, where
40
00:03:04,660 --> 00:03:09,460
people just pull in modules from elsewhere
41
00:03:09,460 --> 00:03:16,540
and just assume that the writer of that
module must know what they are doing,
42
00:03:16,540 --> 00:03:20,650
though without any foundation in reality.
And it is often not the case. Instead,
43
00:03:20,650 --> 00:03:25,480
they are people like you and me, that
also worked exploratively.
44
00:03:25,480 --> 00:03:30,070
You can also do a little thought
experiment and get to this
45
00:03:30,070 --> 00:03:34,210
conclusion yourself; you could even
observe it happening. Let's assume
46
00:03:34,210 --> 00:03:37,840
that somebody finds a better way to
deal with complexity. For example
47
00:03:37,840 --> 00:03:41,200
modularization, or object-oriented
programming, when this was new.
48
00:03:41,200 --> 00:03:44,770
So then you would hope that we would
improve the software that we
49
00:03:44,770 --> 00:03:47,560
had written before, because we now
have it more under control.
50
00:03:47,560 --> 00:03:51,100
But this does not happen.
Instead, we now write bigger
51
00:03:51,100 --> 00:03:57,220
software and work at our limit
again. I think this is not
52
00:03:57,220 --> 00:04:00,400
a problem of software development or
programming, but generally
53
00:04:00,400 --> 00:04:03,790
a problem of humans. Evolution
made us this way, and we have to
54
00:04:03,790 --> 00:04:07,820
learn to deal with it. Let me illustrate
this: I have a theory,
55
00:04:07,820 --> 00:04:14,870
which I call the gradient-theory. The
thesis is, that humans treat their
56
00:04:14,870 --> 00:04:18,110
environment like a process of optimization
in mathematics. This means you
57
00:04:18,110 --> 00:04:22,850
have a terrain and you are looking for
the highest or lowest point - that is an
58
00:04:22,850 --> 00:04:29,360
optimization problem. And you can't
directly aim for it, because you don't
59
00:04:29,360 --> 00:04:34,280
know the terrain. Instead, you have to
make assumptions, and you can observe
60
00:04:34,280 --> 00:04:37,490
this on yourself. If it's too cold, then
you go to the radiator and
61
00:04:37,490 --> 00:04:41,510
you don't set it to the perfect heat,
you set it to "hot", then you wait
62
00:04:41,510 --> 00:04:44,390
until it's too hot, then you
turn it down again.
63
00:04:44,390 --> 00:04:47,510
So we interact with our environment in
a process of approximation.
64
00:04:47,510 --> 00:04:50,030
And not just with heaters, but also when
driving a car, when we have a map.
65
00:04:50,030 --> 00:04:53,840
We look, "where is the limit? Where do
we have to turn?", and
66
00:04:53,840 --> 00:04:58,730
we ignore the journey to the turn,
even if it is nice.
67
00:04:58,730 --> 00:05:03,410
Many things that we do, also including
our choice of speed, is such a gradient.
68
00:05:03,410 --> 00:05:06,320
We accelerate until we feel unwell,
then we slow down again.
69
00:05:06,320 --> 00:05:11,390
Or when searching for something in
a telephone book or dictionary,
70
00:05:11,390 --> 00:05:15,785
we make an assumption of where
it will be. And when it is
71
00:05:15,785 --> 00:05:19,070
too far, we go back again. The essence
of it is: We make an assumption
72
00:05:19,070 --> 00:05:22,580
about what the terrain looks like.
We have smooth transitions here,
73
00:05:22,580 --> 00:05:26,480
so this technique works well.
This is called gradient descent
74
00:05:26,480 --> 00:05:29,930
by the way, when you try to follow
gravity to find the lowest point.
75
00:05:29,930 --> 00:05:34,490
But it does not work well
in two scenarios:
76
00:05:34,490 --> 00:05:38,090
Firstly, when there is a cliff where I
can't go back once I have walked
77
00:05:38,090 --> 00:05:41,930
over it. It also doesn't go well when
you don't notice that you have gone
78
00:05:41,930 --> 00:05:46,400
too far. Well it is similar to the cliff,
and the second problem is
79
00:05:46,400 --> 00:05:49,970
when you can't roll back
for other reasons.
80
00:05:49,970 --> 00:05:53,810
This happens frequently in software
development, and it turns out, that
81
00:05:53,810 --> 00:05:58,340
this is exactly the kind of problem that
human have. For example,
82
00:05:58,340 --> 00:06:03,430
when we have a trial subscription for two
weeks, people forget to cancel it again,
83
00:06:04,030 --> 00:06:09,580
or drug addiction is a classic, or
gambling addiction. And in software
84
00:06:09,580 --> 00:06:12,370
development or project management
in general this is common:
85
00:06:12,370 --> 00:06:17,260
We have already invested so much that
we can't go back. Security is not
86
00:06:17,260 --> 00:06:22,240
a gradient. It may look like one, but it
isn't. I think this is
87
00:06:22,240 --> 00:06:26,800
a fundamental issue in IT security.
You don't notice when you
88
00:06:26,800 --> 00:06:30,640
have gone too far. You only notice
when you get hacked. And then
89
00:06:30,640 --> 00:06:35,020
you can no longer go back, all the data
is already gone. Complexity is also
90
00:06:35,020 --> 00:06:38,260
not a gradient, similarly to security,
but it feels like one. I think
91
00:06:38,260 --> 00:06:42,130
this is the reason why we deal with
it so badly. It feels
92
00:06:42,130 --> 00:06:45,130
as if we have everything under
control. And when we notice,
93
00:06:45,130 --> 00:06:50,140
that we don't, we can't go back.
By the way, giving out data to
94
00:06:50,140 --> 00:06:54,820
facebook is also such a "pseudo-gradient".
95
00:06:54,820 --> 00:07:00,550
When you notice that you gave away too
much, it is too late.
96
00:07:00,550 --> 00:07:05,650
So the conclusion is:
Complexity is evil. We notice it too
97
00:07:05,650 --> 00:07:09,610
late and we get into it too easily.
So we have to counteract that somehow.
98
00:07:09,610 --> 00:07:14,680
If this is our job, we are externalizing
the costs to our customers,
99
00:07:14,680 --> 00:07:19,480
to our users, and to our future self.
100
00:07:19,480 --> 00:07:24,700
This is why you rarely find older software
developers that are happy.
101
00:07:24,700 --> 00:07:28,901
Audience laughs
So, this was the first train of thought,
102
00:07:28,901 --> 00:07:32,786
that led me in this direction. The second
train of thought: Let me just show you
103
00:07:32,786 --> 00:07:35,854
the GNU manifesto, as a representative.
This is not GNU-bashing,
104
00:07:35,854 --> 00:07:39,484
but you can show this pretty well with
the example of the GNU manifesto.
105
00:07:39,484 --> 00:07:43,647
This is the original announcement of the
GNU project by Richard Stallman. He wrote:
106
00:07:43,647 --> 00:07:47,939
"GNU will be able to run Unix programs, but
will not be identical to Unix. We will make
107
00:07:47,939 --> 00:07:53,041
all improvements that are convenient".
This is a very bad sentence.
108
00:07:53,041 --> 00:07:58,473
What does "convenient" mean? For who?
109
00:07:58,473 --> 00:08:03,258
But this is the approach that a lot of
programmers have:
110
00:08:03,258 --> 00:08:07,281
"Oh we can just add this quickly."
We are lacking a corrective, that
111
00:08:07,281 --> 00:08:11,304
we think in advance "what legacy am I
hanging to my leg right now?"
112
00:08:11,304 --> 00:08:15,766
I think this "convenience" thought when
extending software is our "original sin"
113
00:08:15,766 --> 00:08:20,010
- to get a bit catholic here -
in software development.
114
00:08:20,010 --> 00:08:24,252
Everyone has done it before, and you
just can't correct it after the fact.
115
00:08:24,252 --> 00:08:27,256
So the only way of getting rid of it
is to throw away
116
00:08:27,256 --> 00:08:31,626
the whole software or module and
start over again. But software doesn't die.
117
00:08:31,626 --> 00:08:36,592
Only when dealing with software, I learned
that it is good that people die,
118
00:08:36,592 --> 00:08:40,508
because it is a corrective that is needed.
If a system is supposed to improve,
119
00:08:40,508 --> 00:08:44,026
the old stuff has to be able to die at
some point. And this does not
120
00:08:44,026 --> 00:08:49,584
happen with software. It is a feature
that things don't last forever.
121
00:08:49,584 --> 00:08:55,269
In general, you can observe that when
somebody is extending their software and
122
00:08:55,269 --> 00:08:58,484
they have a choice between "We do
something to solve our specific problem"
123
00:08:58,484 --> 00:09:01,905
or "We do something to solve a more
general problem", people will
124
00:09:01,905 --> 00:09:06,636
always try to solve the
more general problem.
125
00:09:06,636 --> 00:09:12,057
"The more danger, the more honor."
And you can see this across the board.
126
00:09:12,057 --> 00:09:16,859
There are very few exceptions to this. And
I had my "aha-moment" when I opened
127
00:09:16,859 --> 00:09:21,215
'gdb' on a project one day. I took '/tmp'
here, but that project was
128
00:09:21,215 --> 00:09:26,135
some checkout.
In my webserver, I have a '.gdbinit' file.
129
00:09:26,135 --> 00:09:30,507
It's a configuration file for the GNU-
debugger, where you can for example say
130
00:09:30,507 --> 00:09:33,405
"Open this application that I want to
131
00:09:33,405 --> 00:09:36,808
debug with these arguments!"
And in there, I write "Don't use Port 80,
132
00:09:36,808 --> 00:09:41,393
that doesn't work, instead use port
8005" or something, to debug it on
133
00:09:41,393 --> 00:09:46,097
localhost. And one day, gdb started
saying "no, I don't accept this
134
00:09:46,097 --> 00:09:50,553
.gdbinit file because it is in a directory
135
00:09:50,553 --> 00:09:56,000
that you have not specifically allowed."
This was exactly such an attempt to fix
136
00:09:56,000 --> 00:10:01,097
an issue after shipping, after the fact.
gdb noticed: "Our config-file has become
137
00:10:01,097 --> 00:10:05,810
so powerful, that it is a security issue",
138
00:10:05,810 --> 00:10:11,038
and then retroactively nailed down the
whole config. And this broke more
139
00:10:11,038 --> 00:10:15,686
than it needed to - perhaps, I don't
know for sure - but it was very annoying
140
00:10:15,686 --> 00:10:19,270
for me. You can put an auto path in here,
but that is when I noticed it
141
00:10:19,270 --> 00:10:22,218
for the first time. This was a few years
142
00:10:22,218 --> 00:10:25,942
ago. I don't know, when exactly that was.
There was a similar case like this
143
00:10:25,942 --> 00:10:30,041
again: With Vim, the editor, that I like
to use. You can do things like
144
00:10:30,041 --> 00:10:33,882
in a comment in the file that is being
edited, you can put some configuration
145
00:10:33,882 --> 00:10:37,028
settings in the first or last three lines.
146
00:10:37,028 --> 00:10:41,870
It is supposed to be used for "I use
tabstop=4 here", or something.
147
00:10:41,870 --> 00:10:46,160
But the parser for this had
a security bug, which made it
148
00:10:46,160 --> 00:10:50,512
possible to create a file that
executes code, when it is
149
00:10:50,512 --> 00:10:55,564
opened in vim, which was of course
not intended. But it is the same
150
00:10:55,564 --> 00:10:59,847
issue. I think you can generalize this
151
00:10:59,847 --> 00:11:03,135
a bit - though earlier I argued
against generalizations, but
152
00:11:03,135 --> 00:11:06,535
in analysis it is good, in software
it is usually bad. let me illustrate
153
00:11:06,535 --> 00:11:10,777
with an example:
Let's assume that we have a CSV file
154
00:11:10,777 --> 00:11:16,194
with some trouble tickets. Field 4
is the one, that we are interested in.
155
00:11:16,194 --> 00:11:21,511
Let's assume it looks like this. It's CSV.
So, now I would like to have the sum
156
00:11:21,511 --> 00:11:26,285
of the four fields. So first I use
cut, we are in Unix here.
157
00:11:26,285 --> 00:11:31,012
Then the first line has to go,
158
00:11:31,012 --> 00:11:34,193
so I use tail. Now the first line
is gone, now I just have to
159
00:11:34,193 --> 00:11:37,746
calculate the sum. There is an
application for this too: paste. that is
160
00:11:37,746 --> 00:11:43,442
how you do it in Unix. Then I have to
calculate it. There we go! But what if
161
00:11:43,442 --> 00:11:49,381
it doesn't say 1 here, but instead "fred"?
We notice: cut does not have a problem,
162
00:11:49,381 --> 00:11:54,442
tail does not have a problem, paste is
fine, but bc falls on its face.
163
00:11:54,442 --> 00:12:01,973
Even worse, bc is programmable.
There could be the
164
00:12:01,973 --> 00:12:05,214
Ackermann-function here and
your computer would be gone
165
00:12:05,214 --> 00:12:09,772
for an hour, while it is trying to
solve some recursion. And I think it
166
00:12:09,772 --> 00:12:14,823
is useful to introduce a concept here
to say: cut, tail and paste are harmless,
167
00:12:14,823 --> 00:12:18,817
bc is not. This is one of the thoughts
where I thought "okay, you can make
168
00:12:18,817 --> 00:12:22,152
a talk about this".
But this is not enough.
169
00:12:22,152 --> 00:12:27,235
There are different kinds of harmless.
But I think this simple idea
170
00:12:27,235 --> 00:12:31,405
already helps us a bit.
Let's make it into a sentence:
171
00:12:31,405 --> 00:12:35,204
Software is harmless, when unexpected
input don't produce unexpected
172
00:12:35,204 --> 00:12:38,868
behavior or unexpected kinds of output.
For example, an SHA-checksum is always
173
00:12:38,868 --> 00:12:43,166
harmless. Regardless of
what data I put in, the output
174
00:12:43,166 --> 00:12:47,742
has a known format. Or word
count (wc) is also one of those.
175
00:12:47,742 --> 00:12:52,104
Now you could say: "Okay, just use
awk!" And in awk I don't have a problem
176
00:12:52,104 --> 00:12:55,955
when it says "fred" instead of "4"
and the interpreter also does not
177
00:12:55,955 --> 00:13:00,541
interpret any functions.
It looks better, but
178
00:13:00,541 --> 00:13:05,397
is it really harmless?
It turns out, awk is a different kind of
179
00:13:05,397 --> 00:13:09,385
not harmless, because you can write
in the filesystem with it. So I don't have
180
00:13:09,385 --> 00:13:13,548
to worry about the input, but I have to
worry about the code, that I hand to it
181
00:13:13,548 --> 00:13:17,275
on the command line. So that is
another distinction you can make.
182
00:13:17,275 --> 00:13:21,812
This is a big problem in the game
industry by the way:
183
00:13:21,812 --> 00:13:25,862
The game development industry
has started putting interpreters
184
00:13:25,862 --> 00:13:30,856
into their games, to be able to write
their business logic - not the AI,
185
00:13:30,856 --> 00:13:36,820
but small scripts - in a scripting
language. One of the most
186
00:13:36,820 --> 00:13:41,132
popular script-interpreters for this
purpose is Lua. And Lua is primarily
187
00:13:41,132 --> 00:13:45,091
used because it can't do anything,
if you don't specifically allow it.
188
00:13:45,091 --> 00:13:48,926
So It can't open files or sockets.
You can enable this manually though,
189
00:13:48,926 --> 00:13:53,190
and then you have a problem again
of course. But this is a real issue.
190
00:13:53,190 --> 00:13:57,149
Many open-source people don't think
about this, because they think "Well,
191
00:13:57,149 --> 00:14:00,358
I will ship it and the rest is no longer
my issue." But I think,
192
00:14:00,358 --> 00:14:03,335
that we generally have to think
about this, and preferably
193
00:14:03,335 --> 00:14:06,771
before shipping, optimally already while
programming. So, this is
194
00:14:06,771 --> 00:14:11,226
a different kind of harmlessness.
The first kind was "Can bad input
195
00:14:11,226 --> 00:14:15,014
cause bad output?" And now: "Can the
application itself do bad things?"
196
00:14:15,014 --> 00:14:19,322
This is a very modern thought,
because we work a lot more with
197
00:14:19,322 --> 00:14:23,874
sandboxing today. In sandboxing, the goal
is to prevent a program from
198
00:14:23,874 --> 00:14:28,024
accidentally or deliberately doing bad
things. And there are again different
199
00:14:28,024 --> 00:14:32,605
things that a program can do.
bc can eat processing time. awk can
200
00:14:32,605 --> 00:14:37,095
read and write in your filesystem, and
this goes on and on. Let's get back
201
00:14:37,095 --> 00:14:41,740
to the GNU manifesto: GNU awk is a special
version of awk and it can open sockets,
202
00:14:41,740 --> 00:14:45,652
without any need. This means, if we
just use awk and thing "Well, awk can
203
00:14:45,652 --> 00:14:49,086
write in the filesystem, but I mounted
that read-only, so nothing
204
00:14:49,086 --> 00:14:53,457
can happen". But then if GNU awk
is being used, it is suddenly
205
00:14:53,457 --> 00:14:57,802
no longer harmless. Bash
can open sockets too by the way!
206
00:14:57,802 --> 00:15:02,788
I don't know, how many people knew that?
This goes on of course: after awk
207
00:15:02,788 --> 00:15:06,446
came Perl. It's even worse, and
Perl can do eval(), which in my
208
00:15:06,446 --> 00:15:11,425
opinion is the worse evil that you can
have in a programming language.
209
00:15:11,425 --> 00:15:15,985
A bit closer to the end-user you can also
observe this in browsers. Let's look at
210
00:15:15,985 --> 00:15:20,523
Netscape for example:
Several times, Netscape had the choice
211
00:15:20,523 --> 00:15:24,977
between "useful" and "harmless" and always
chose "useful". It started with
212
00:15:24,977 --> 00:15:29,442
the plugins. I don't know, who
of you still remembers the Flash-plugin,
213
00:15:29,442 --> 00:15:33,755
or before that we all had the RealPlayer,
and there was also an Acrobat-plugin -
214
00:15:33,755 --> 00:15:37,641
And all of it was shit, because the
plugins were native code: they could do
215
00:15:37,641 --> 00:15:41,829
everything, that their operating system
allowed. This means that it was very
216
00:15:41,829 --> 00:15:45,635
useful, but also very dangerous.
And it was a conscious choice of
217
00:15:45,635 --> 00:15:49,579
the browsers, to allow this.
The actual goal of this talk is
218
00:15:49,579 --> 00:15:54,202
to give the programmers among you a
bit of awareness that you don't just
219
00:15:54,202 --> 00:15:58,933
add a plugin interface that
can do everything.
220
00:15:58,933 --> 00:16:04,564
The next iteration was:
We'll do everything in JavaScript.
221
00:16:04,564 --> 00:16:09,562
At first it looked better, but this
JavaScript eventually also ran with
222
00:16:09,562 --> 00:16:13,861
enough privileges do do bad things
in the system, or at least in the browser.
223
00:16:13,861 --> 00:16:17,610
It turns out: People now have their
important data in the browser,
224
00:16:17,610 --> 00:16:21,064
because they do online banking. And
that is enough do do a lot of damage.
225
00:16:21,064 --> 00:16:25,609
Then they had to correct it
after the fact. Chrome now imposes
226
00:16:25,609 --> 00:16:29,383
even further limits for security reasons
to break ad blockers. It's always
227
00:16:29,383 --> 00:16:32,601
the same trap that we walk into.
Who of you here use Windows?
228
00:16:32,601 --> 00:16:37,285
In Windows there is a tool by
Mark Russinovich - by now he has
229
00:16:37,285 --> 00:16:41,300
sold it to Microsoft, so it is now an
official Microsoft tool.
230
00:16:41,300 --> 00:16:44,680
And the only functionality of this
tool is to list the different
231
00:16:44,680 --> 00:16:48,013
plugins that are part of the system.
And I took a relatively
232
00:16:48,013 --> 00:16:52,285
clean system here. It's not about
this down here or
233
00:16:52,285 --> 00:16:56,549
the size of the scrollbar, but just
how many tabs there are at the top:
234
00:16:56,549 --> 00:17:00,745
These are all different options for
plugins to integrate into the system,
235
00:17:00,745 --> 00:17:04,445
and nobody has an overview of this
anymore, because people always decided
236
00:17:04,445 --> 00:17:08,798
to go in the wrong direction. I believe
that this is a core problem.
237
00:17:08,798 --> 00:17:13,857
There is a third approach to this:
My daily life in security consists of
238
00:17:13,857 --> 00:17:17,926
going to companies. They show me their
source code and I look for bugs. Then
239
00:17:17,926 --> 00:17:21,920
I tell them, which bugs I found. And
occasionally, there are cases where
240
00:17:21,920 --> 00:17:25,808
I notice that there are a lot of bugs.
Not just those that I find, but they
241
00:17:25,808 --> 00:17:30,035
already have their own database,
a bugtracker, and they already
242
00:17:30,035 --> 00:17:34,955
have a seven-digit number of bugs. Yes,
This happens. And since it is a problem
243
00:17:34,955 --> 00:17:39,361
that we have so many bugs, there
are now counter-strategies by developers
244
00:17:39,361 --> 00:17:42,746
that start saying: "Okay, if this bug is
not important then
245
00:17:42,746 --> 00:17:46,830
I can fix it later." And "later" means
"never" in reality. It just sits there.
246
00:17:46,830 --> 00:17:52,134
Joke that only makes sense in German
247
00:17:52,134 --> 00:17:58,087
In the real world, bug
trackers are often just
248
00:17:58,087 --> 00:18:03,812
massive permanent data disposal sites:
For example, I recently filed a bug report
249
00:18:03,812 --> 00:18:08,146
for Firefox and got the ID 1590000.
This is already a bad sign.
250
00:18:08,146 --> 00:18:11,876
But it is also a good sign, that
the bug tracker is open.
251
00:18:11,876 --> 00:18:16,007
For Microsoft you can't see how
many bugs they have.
252
00:18:16,007 --> 00:18:19,501
This is only meant for illustration.
Mozilla is not especially bad.
253
00:18:19,501 --> 00:18:23,170
Mozilla just has an open tracker,
on which I can show it well.
254
00:18:23,170 --> 00:18:27,217
What I wanted to show you -
I had a look: "What is the first bug
255
00:18:27,217 --> 00:18:31,017
that I filed there?" It still had
a six-digit ID.
256
00:18:31,017 --> 00:18:37,953
That was 2003. If you look at the
history of bug IDs then you notice:
257
00:18:37,953 --> 00:18:43,047
It is growing exponentially.
And it's not like the bugs somehow
258
00:18:43,047 --> 00:18:48,431
go away at some point.
I have noticed two major events,
259
00:18:48,431 --> 00:18:52,235
where bugs are closed:
When a new release is done
260
00:18:52,235 --> 00:18:55,851
and you throw out the old JavaScript
engine and put in a new one.
261
00:18:55,851 --> 00:18:59,700
Then you just close all bugs of the old
engine. It looks as if you have achieved
262
00:18:59,700 --> 00:19:03,568
something. And the second is this one:
I don't know, can you read this in
263
00:19:03,568 --> 00:19:06,848
the back? Mozilla just closed my bug.
It says:
264
00:19:06,848 --> 00:19:10,034
"This bug has been automatically
resolved after a period
265
00:19:10,034 --> 00:19:14,008
of inactivity". Mind you, it was not me
who was inactive. I filed the bug and
266
00:19:14,008 --> 00:19:17,750
nobody at Mozilla took care of it.
So they just automatically closed it,
267
00:19:17,750 --> 00:19:21,355
because the statistics look so bad.
This is a big issue,
268
00:19:21,355 --> 00:19:24,378
not just at Mozilla. As I said, this is
just the example
269
00:19:24,378 --> 00:19:28,262
that I can show, because
in their case it is public. But
270
00:19:28,262 --> 00:19:32,349
this leads to a cascade of action
and reaction. For example,
271
00:19:32,349 --> 00:19:36,089
unimportant bugs are just not fixed
anymore. And then people
272
00:19:36,089 --> 00:19:39,461
add "important" on their bugs,
because they want them to be fixed.
273
00:19:39,461 --> 00:19:42,780
Then they say "Okay, the important
bugs also don't get fixed,
274
00:19:42,780 --> 00:19:46,849
because there are too many of them."
And then people
275
00:19:46,849 --> 00:19:51,472
write "Security" on their bugs, and now
we have a wave of security-bugs.
276
00:19:51,472 --> 00:19:56,008
There they negotiate: "Is this really
a problem?" And then we get excuses
277
00:19:56,008 --> 00:20:01,232
like "It's just a crash."
The point is that there is an unholy
278
00:20:01,232 --> 00:20:07,589
alliance with another trend,
namely that companies see:
279
00:20:07,589 --> 00:20:11,476
We have so many bugs open that
solving the bugs is not the goal anymore.
280
00:20:11,476 --> 00:20:15,295
There are just too many, it is
unrealistic. Instead,
281
00:20:15,295 --> 00:20:19,598
we introduce metrics like "we do
fuzzing". Fuzzing is not
282
00:20:19,598 --> 00:20:23,897
a bad idea, but it is not "finding all
bugs", but just the first step
283
00:20:23,897 --> 00:20:28,090
on a long road. But it gives
out a nice metric.
284
00:20:28,090 --> 00:20:33,011
We have so-and-so many fuzz-
testcases, and now...
285
00:20:33,011 --> 00:20:37,402
Are we now better or worse than
before? It's hard to say.
286
00:20:37,402 --> 00:20:41,769
287
00:20:41,769 --> 00:20:46,975
288
00:20:46,975 --> 00:20:51,635
289
00:20:51,635 --> 00:20:55,373
290
00:20:55,373 --> 00:20:58,367
291
00:20:58,367 --> 00:21:01,752
292
00:21:01,752 --> 00:21:05,930
293
00:21:05,930 --> 00:21:09,778
294
00:21:09,778 --> 00:21:13,974
295
00:21:13,974 --> 00:21:18,438
296
00:21:18,438 --> 00:21:20,345
297
00:21:20,345 --> 00:21:23,400
298
00:21:23,400 --> 00:21:28,686
299
00:21:28,686 --> 00:21:33,658
300
00:21:33,658 --> 00:21:37,336
301
00:21:37,336 --> 00:21:41,931
302
00:21:41,931 --> 00:21:47,848
303
00:21:47,848 --> 00:21:51,779
304
00:21:51,779 --> 00:21:55,288
305
00:21:55,288 --> 00:21:58,910
306
00:21:58,910 --> 00:22:03,159
307
00:22:03,159 --> 00:22:07,913
308
00:22:07,913 --> 00:22:12,536
309
00:22:12,536 --> 00:22:17,253
310
00:22:17,253 --> 00:22:21,690
311
00:22:21,690 --> 00:22:26,237
312
00:22:26,237 --> 00:22:30,208
313
00:22:30,208 --> 00:22:33,999
314
00:22:33,999 --> 00:22:36,695
315
00:22:36,695 --> 00:22:41,009
316
00:22:41,009 --> 00:22:45,459
317
00:22:45,459 --> 00:22:49,214
318
00:22:49,214 --> 00:22:52,446
319
00:22:52,446 --> 00:22:56,310
320
00:22:56,310 --> 00:23:00,526
321
00:23:00,526 --> 00:23:04,246
322
00:23:04,246 --> 00:23:07,796
323
00:23:07,796 --> 00:23:12,318
324
00:23:12,318 --> 00:23:16,240
325
00:23:16,240 --> 00:23:20,475
326
00:23:20,475 --> 00:23:24,298
327
00:23:24,298 --> 00:23:27,676
328
00:23:27,676 --> 00:23:31,460
329
00:23:31,460 --> 00:23:36,410
330
00:23:36,410 --> 00:23:41,173
331
00:23:41,173 --> 00:23:45,020
332
00:23:45,020 --> 00:23:48,800
333
00:23:48,800 --> 00:23:53,213
334
00:23:53,213 --> 00:23:57,578
335
00:23:57,578 --> 00:24:03,977
336
00:24:03,977 --> 00:24:10,106
337
00:24:10,106 --> 00:24:13,625
338
00:24:13,625 --> 00:24:17,247
339
00:24:17,247 --> 00:24:21,130
340
00:24:21,130 --> 00:24:24,965
341
00:24:24,965 --> 00:24:28,860
342
00:24:28,860 --> 00:24:33,035
343
00:24:33,035 --> 00:24:37,879
344
00:24:37,879 --> 00:24:42,818
345
00:24:42,818 --> 00:24:48,100
346
00:24:48,100 --> 00:24:52,500
347
00:24:52,500 --> 00:24:56,206
348
00:24:56,206 --> 00:24:59,868
349
00:24:59,868 --> 00:25:03,716
350
00:25:03,716 --> 00:25:08,630
351
00:25:08,630 --> 00:25:13,261
352
00:25:13,261 --> 00:25:16,930
353
00:25:16,930 --> 00:25:19,907
354
00:25:19,907 --> 00:25:23,917
355
00:25:23,917 --> 00:25:27,947
356
00:25:27,947 --> 00:25:31,971
357
00:25:31,971 --> 00:25:35,440
358
00:25:35,440 --> 00:25:39,425
359
00:25:39,425 --> 00:25:44,229
360
00:25:44,229 --> 00:25:48,829
361
00:25:48,829 --> 00:25:52,582
362
00:25:52,582 --> 00:25:57,265
363
00:25:57,265 --> 00:26:02,796
364
00:26:02,796 --> 00:26:08,344
365
00:26:08,344 --> 00:26:13,337
366
00:26:13,337 --> 00:26:16,676
367
00:26:16,676 --> 00:26:19,707
368
00:26:19,707 --> 00:26:23,220
369
00:26:23,220 --> 00:26:26,838
370
00:26:26,838 --> 00:26:30,901
371
00:26:30,901 --> 00:26:34,758
372
00:26:34,758 --> 00:26:38,616
373
00:26:38,616 --> 00:26:42,095
374
00:26:42,095 --> 00:26:46,015
375
00:26:46,015 --> 00:26:49,071
376
00:26:49,071 --> 00:26:53,416
377
00:26:53,416 --> 00:26:57,395
378
00:26:57,395 --> 00:27:01,075
379
00:27:01,075 --> 00:27:03,522
380
00:27:03,522 --> 00:27:06,250
381
00:27:06,250 --> 00:27:09,796
382
00:27:09,796 --> 00:27:13,000
383
00:27:13,897 --> 00:27:17,249
384
00:27:17,249 --> 00:27:22,275
385
00:27:22,275 --> 00:27:26,830
386
00:27:26,830 --> 00:27:29,650
387
00:27:29,650 --> 00:27:33,349
388
00:27:33,349 --> 00:27:37,155
389
00:27:37,155 --> 00:27:41,157
390
00:27:41,157 --> 00:27:46,006
391
00:27:46,006 --> 00:27:49,877
392
00:27:49,877 --> 00:27:53,429
393
00:27:53,429 --> 00:27:56,744
394
00:27:56,744 --> 00:28:00,948
395
00:28:00,948 --> 00:28:05,771
396
00:28:05,771 --> 00:28:10,095
397
00:28:10,095 --> 00:28:14,460
398
00:28:14,460 --> 00:28:19,729
399
00:28:19,729 --> 00:28:24,462
400
00:28:24,462 --> 00:28:28,774
401
00:28:28,774 --> 00:28:33,027
402
00:28:33,027 --> 00:28:38,276
403
00:28:38,276 --> 00:28:42,951
404
00:28:42,951 --> 00:28:46,781
405
00:28:46,781 --> 00:28:50,708
406
00:28:50,708 --> 00:28:54,292
407
00:28:54,292 --> 00:28:58,676
408
00:28:58,676 --> 00:29:03,695
409
00:29:03,695 --> 00:29:07,750
410
00:29:07,750 --> 00:29:10,944
411
00:29:10,944 --> 00:29:14,235
412
00:29:14,235 --> 00:29:18,170
413
00:29:18,170 --> 00:29:21,906
414
00:29:21,906 --> 00:29:25,648
415
00:29:25,648 --> 00:29:29,813
416
00:29:29,813 --> 00:29:33,962
417
00:29:33,962 --> 00:29:37,258
418
00:29:37,258 --> 00:29:40,789
419
00:29:40,789 --> 00:29:44,961
420
00:29:44,961 --> 00:29:49,101
421
00:29:49,101 --> 00:29:53,353
422
00:29:53,353 --> 00:29:58,053
423
00:29:58,053 --> 00:30:02,925
424
00:30:02,925 --> 00:30:07,068
425
00:30:07,068 --> 00:30:11,207
426
00:30:11,207 --> 00:30:15,768
427
00:30:15,768 --> 00:30:20,235
428
00:30:20,235 --> 00:30:24,736
429
00:30:24,736 --> 00:30:28,717
430
00:30:28,717 --> 00:30:33,343
431
00:30:33,343 --> 00:30:38,565
432
00:30:38,565 --> 00:30:43,107
433
00:30:43,107 --> 00:30:49,769
434
00:30:49,769 --> 00:30:55,467
435
00:30:55,467 --> 00:30:59,760
436
00:30:59,760 --> 00:31:04,521
437
00:31:04,521 --> 00:31:08,482
438
00:31:08,482 --> 00:31:12,363
439
00:31:12,363 --> 00:31:18,015
440
00:31:18,015 --> 00:31:24,111
441
00:31:24,111 --> 00:31:29,850
442
00:31:29,850 --> 00:31:36,526
443
00:31:36,526 --> 00:31:42,156
444
00:31:42,156 --> 00:31:46,170
445
00:31:46,170 --> 00:31:49,830
446
00:31:49,830 --> 00:31:53,224
447
00:31:53,224 --> 00:31:56,585
448
00:31:56,585 --> 00:32:00,581
449
00:32:00,581 --> 00:32:05,106
450
00:32:05,106 --> 00:32:08,988
451
00:32:08,988 --> 00:32:12,573
452
00:32:12,573 --> 00:32:16,852
453
00:32:16,852 --> 00:32:21,549
454
00:32:21,549 --> 00:32:24,654
455
00:32:24,654 --> 00:32:28,779
456
00:32:28,779 --> 00:32:34,307
457
00:32:34,307 --> 00:32:39,450
458
00:32:39,450 --> 00:32:44,572
459
00:32:44,572 --> 00:32:49,299
460
00:32:49,299 --> 00:32:52,794
461
00:32:52,794 --> 00:32:56,071
462
00:32:56,071 --> 00:32:59,467
463
00:32:59,467 --> 00:33:03,335
464
00:33:03,335 --> 00:33:08,721
465
00:33:08,721 --> 00:33:13,595
466
00:33:13,595 --> 00:33:17,871
467
00:33:17,871 --> 00:33:22,411
468
00:33:22,411 --> 00:33:26,844
469
00:33:26,844 --> 00:33:31,971
470
00:33:31,971 --> 00:33:37,346
471
00:33:37,346 --> 00:33:41,891
472
00:33:41,891 --> 00:33:47,106
473
00:33:47,106 --> 00:33:52,356
474
00:33:52,356 --> 00:33:56,950
475
00:33:56,950 --> 00:34:01,703
476
00:34:01,703 --> 00:34:05,389
477
00:34:05,389 --> 00:34:10,021
478
00:34:10,021 --> 00:34:15,049
479
00:34:15,049 --> 00:34:20,110
480
00:34:20,110 --> 00:34:25,645
481
00:34:25,645 --> 00:34:29,835
482
00:34:29,835 --> 00:34:34,685
483
00:34:34,685 --> 00:34:39,107
484
00:34:39,107 --> 00:34:43,660
485
00:34:43,660 --> 00:34:48,532
486
00:34:48,532 --> 00:34:54,215
487
00:34:54,215 --> 00:34:59,203
488
00:34:59,203 --> 00:35:02,580
489
00:35:02,580 --> 00:35:07,256
490
00:35:07,256 --> 00:35:11,814
491
00:35:11,814 --> 00:35:16,713
492
00:35:16,713 --> 00:35:21,634
493
00:35:21,634 --> 00:35:25,555
494
00:35:25,555 --> 00:35:30,453
495
00:35:30,453 --> 00:35:34,457
496
00:35:34,457 --> 00:35:39,444
497
00:35:39,444 --> 00:35:44,786
498
00:35:44,786 --> 00:35:50,341
499
00:35:50,341 --> 00:35:55,009
500
00:35:55,009 --> 00:35:59,622
501
00:35:59,622 --> 00:36:04,077
502
00:36:04,077 --> 00:36:07,868
503
00:36:07,868 --> 00:36:11,956
504
00:36:11,956 --> 00:36:16,785
505
00:36:16,785 --> 00:36:21,632
506
00:36:21,632 --> 00:36:26,415
507
00:36:26,415 --> 00:36:30,549
508
00:36:30,549 --> 00:36:34,335
509
00:36:34,335 --> 00:36:38,224
510
00:36:38,224 --> 00:36:41,825
511
00:36:41,825 --> 00:36:46,801
512
00:36:46,801 --> 00:36:53,313
513
00:36:53,313 --> 00:36:58,917
514
00:36:58,917 --> 00:37:04,896
515
00:37:04,896 --> 00:37:09,333
516
00:37:09,333 --> 00:37:12,456
517
00:37:12,456 --> 00:37:15,758
518
00:37:15,758 --> 00:37:19,001
519
00:37:19,001 --> 00:37:23,259
520
00:37:23,259 --> 00:37:26,800
521
00:37:26,800 --> 00:37:30,960
522
00:37:30,960 --> 00:37:35,492
523
00:37:35,492 --> 00:37:39,209
524
00:37:39,209 --> 00:37:42,595
525
00:37:42,595 --> 00:37:47,263
526
00:37:47,263 --> 00:37:52,198
527
00:37:52,198 --> 00:37:55,821
528
00:37:55,821 --> 00:37:59,210
529
00:37:59,210 --> 00:38:02,628
530
00:38:02,628 --> 00:38:06,296
531
00:38:06,296 --> 00:38:10,513
532
00:38:10,513 --> 00:38:14,693
533
00:38:14,693 --> 00:38:20,115
534
00:38:20,115 --> 00:38:25,190
535
00:38:25,190 --> 00:38:28,810
536
00:38:28,810 --> 00:38:32,660
537
00:38:32,660 --> 00:38:36,058
538
00:38:36,058 --> 00:38:40,541
539
00:38:40,541 --> 00:38:45,744
540
00:38:45,744 --> 00:38:50,294
541
00:38:50,294 --> 00:38:55,405
542
00:38:55,405 --> 00:39:00,857
543
00:39:00,857 --> 00:39:04,590
544
00:39:04,590 --> 00:39:07,374
545
00:39:07,374 --> 00:39:11,371
546
00:39:11,371 --> 00:39:16,050
547
00:39:16,050 --> 00:39:20,072
548
00:39:20,072 --> 00:39:24,693
549
00:39:24,693 --> 00:39:29,356
550
00:39:29,356 --> 00:39:33,258
551
00:39:33,258 --> 00:39:36,727
552
00:39:36,727 --> 00:39:40,252
553
00:39:40,252 --> 00:39:42,612
554
00:39:42,612 --> 00:39:45,735
555
00:39:45,735 --> 00:39:51,611
556
00:39:51,611 --> 00:39:56,651
557
00:39:56,651 --> 00:40:00,428
558
00:40:00,428 --> 00:40:03,856
559
00:40:03,856 --> 00:40:07,917
560
00:40:07,917 --> 00:40:11,881
561
00:40:11,881 --> 00:40:15,607
562
00:40:15,607 --> 00:40:19,101
563
00:40:19,101 --> 00:40:23,064
564
00:40:23,064 --> 00:40:27,135
565
00:40:27,135 --> 00:40:31,788
566
00:40:31,788 --> 00:40:36,450
567
00:40:36,450 --> 00:40:40,591
568
00:40:40,591 --> 00:40:45,691
569
00:40:45,691 --> 00:40:49,752
570
00:40:49,752 --> 00:40:53,954
571
00:40:53,954 --> 00:40:58,110
572
00:40:58,110 --> 00:41:03,000
573
00:41:03,000 --> 00:41:06,210
574
00:41:06,210 --> 00:41:11,460
575
00:41:11,460 --> 00:41:14,430
576
00:41:14,430 --> 00:41:19,740
577
00:41:19,740 --> 00:41:23,460
578
00:41:23,460 --> 00:41:29,310
579
00:41:29,310 --> 00:41:32,400
580
00:41:32,400 --> 00:41:40,080
581
00:41:40,080 --> 00:41:44,070
582
00:41:44,070 --> 00:41:48,300
583
00:41:49,020 --> 00:41:54,660
584
00:41:54,660 --> 00:42:01,480
585
00:42:01,480 --> 00:42:04,990
586
00:42:04,990 --> 00:42:08,170
587
00:42:08,170 --> 00:42:12,070
588
00:42:12,070 --> 00:42:16,060
589
00:42:16,060 --> 00:42:19,240
590
00:42:19,240 --> 00:42:25,810
591
00:42:25,810 --> 00:42:29,020
592
00:42:29,020 --> 00:42:33,160
593
00:42:33,160 --> 00:42:39,310
594
00:42:39,310 --> 00:42:42,040
595
00:42:42,040 --> 00:42:44,950
596
00:42:44,950 --> 00:42:48,070
597
00:42:48,070 --> 00:42:51,880
598
00:42:51,880 --> 00:42:55,120
599
00:42:55,120 --> 00:42:58,420
600
00:42:58,420 --> 00:43:05,530
601
00:43:05,530 --> 00:43:08,680
602
00:43:08,680 --> 00:43:13,150
603
00:43:13,150 --> 00:43:17,980
604
00:43:17,980 --> 00:43:23,380
605
00:43:23,380 --> 00:43:26,740
606
00:43:26,740 --> 00:43:31,570
607
00:43:31,570 --> 00:43:35,260
608
00:43:35,260 --> 00:43:39,820
609
00:43:39,820 --> 00:43:43,420
610
00:43:43,420 --> 00:43:47,980
611
00:43:47,980 --> 00:43:52,210
612
00:43:52,210 --> 00:43:55,090
613
00:43:55,090 --> 00:43:58,570
614
00:43:58,570 --> 00:44:01,930
615
00:44:01,930 --> 00:44:08,480
616
00:44:08,480 --> 00:44:13,310
617
00:44:13,310 --> 00:44:17,990
618
00:44:17,990 --> 00:44:22,460
619
00:44:22,460 --> 00:44:28,370
620
00:44:28,370 --> 00:44:31,370
621
00:44:31,370 --> 00:44:34,700
622
00:44:34,700 --> 00:44:38,870
623
00:44:38,870 --> 00:44:42,320
624
00:44:42,320 --> 00:44:44,480
625
00:44:44,480 --> 00:44:48,500
626
00:44:48,500 --> 00:44:53,810
627
00:44:53,810 --> 00:44:56,360
628
00:44:56,360 --> 00:44:59,630
629
00:44:59,630 --> 00:45:03,440
630
00:45:03,440 --> 00:45:07,940
631
00:45:07,940 --> 00:45:11,570
632
00:45:11,570 --> 00:45:15,770
633
00:45:15,770 --> 00:45:21,800
634
00:45:21,800 --> 00:45:26,690
635
00:45:26,690 --> 00:45:29,960
636
00:45:29,960 --> 00:45:33,770
637
00:45:33,770 --> 00:45:37,070
638
00:45:37,070 --> 00:45:41,210
639
00:45:41,210 --> 00:45:51,530
640
00:45:51,530 --> 00:45:54,320
641
00:45:54,320 --> 00:45:59,720
642
00:45:59,720 --> 00:46:05,150
643
00:46:05,150 --> 00:46:08,750
644
00:46:08,750 --> 00:46:13,080
645
00:46:13,080 --> 00:46:19,020
646
00:46:19,020 --> 00:46:22,200
647
00:46:22,200 --> 00:46:26,130
648
00:46:26,130 --> 00:46:30,780
649
00:46:30,780 --> 00:46:35,250
650
00:46:35,250 --> 00:46:38,490
651
00:46:38,490 --> 00:46:42,990
652
00:46:44,340 --> 00:46:48,390
653
00:46:48,390 --> 00:46:51,420
654
00:46:51,420 --> 00:46:56,460
655
00:46:56,460 --> 00:47:00,480
656
00:47:00,480 --> 00:47:05,850
657
00:47:05,850 --> 00:47:09,000
658
00:47:09,000 --> 00:47:14,670
659
00:47:14,670 --> 00:47:18,810
660
00:47:18,810 --> 00:47:24,090
661
00:47:24,090 --> 00:47:28,920
662
00:47:28,920 --> 00:47:33,540
663
00:47:33,540 --> 00:47:37,470
664
00:47:37,470 --> 00:47:42,540
665
00:47:42,540 --> 00:47:48,070
666
00:47:48,070 --> 00:47:51,232
667
00:47:51,232 --> 00:47:53,575
668
00:47:53,575 --> 00:47:59,176
669
00:47:59,176 --> 00:48:06,097
670
00:48:06,097 --> 00:48:10,152
671
00:48:10,152 --> 00:48:14,255
672
00:48:14,255 --> 00:48:18,012
673
00:48:18,012 --> 00:48:22,567
674
00:48:22,567 --> 00:48:27,432
675
00:48:27,432 --> 00:48:31,606
676
00:48:31,606 --> 00:48:36,538
677
00:48:36,538 --> 00:48:42,133
678
00:48:42,133 --> 00:48:46,668
679
00:48:46,668 --> 00:48:51,040
680
00:48:51,040 --> 00:48:55,159
681
00:48:55,159 --> 00:48:59,743
682
00:48:59,743 --> 00:49:04,345
683
00:49:04,345 --> 00:49:08,893
684
00:49:08,893 --> 00:49:13,706
685
00:49:13,706 --> 00:49:17,203
686
00:49:17,203 --> 00:49:20,885
687
00:49:20,885 --> 00:49:26,271
688
00:49:26,271 --> 00:49:31,933
689
00:49:31,933 --> 00:49:36,007
690
00:49:36,007 --> 00:49:39,835
691
00:49:39,835 --> 00:49:44,031
692
00:49:44,031 --> 00:49:49,421
693
00:49:49,421 --> 00:49:54,225
694
00:49:54,225 --> 00:49:57,723
695
00:49:57,723 --> 00:50:01,824
696
00:50:01,824 --> 00:50:06,636
697
00:50:06,636 --> 00:50:10,657
698
00:50:10,657 --> 00:50:15,330
699
00:50:15,330 --> 00:50:20,768
700
00:50:20,768 --> 00:50:25,259
701
00:50:25,259 --> 00:50:29,070
702
00:50:29,070 --> 00:50:33,399
703
00:50:33,399 --> 00:50:38,295
704
00:50:38,295 --> 00:50:43,166
705
00:50:43,166 --> 00:50:47,637
706
00:50:47,637 --> 00:50:51,557
707
00:50:51,557 --> 00:50:56,079
708
00:50:56,079 --> 00:50:59,108
709
00:51:00,042 --> 00:51:05,852
710
00:51:05,852 --> 00:51:09,893
711
00:51:09,893 --> 00:51:14,282
712
00:51:14,282 --> 00:51:18,240
713
00:51:18,240 --> 00:51:22,553
714
00:51:22,553 --> 00:51:26,820
715
00:51:26,820 --> 00:51:30,654
716
00:51:30,654 --> 00:51:36,671
717
00:51:36,671 --> 00:51:41,510
718
00:51:41,510 --> 00:51:46,426
719
00:51:46,426 --> 00:51:49,610
720
00:51:49,610 --> 00:51:53,647
721
00:51:53,647 --> 00:51:58,069
722
00:51:58,069 --> 00:52:09,897
723
00:52:10,565 --> 00:52:13,514
724
00:52:13,514 --> 00:52:16,685
725
00:52:16,685 --> 00:52:20,779
726
00:52:20,779 --> 00:52:24,503
727
00:52:24,503 --> 00:52:30,910
728
00:52:30,910 --> 00:52:34,660
729
00:52:34,660 --> 00:52:40,990
730
00:52:40,990 --> 00:52:44,440
731
00:52:44,440 --> 00:52:47,110
732
00:52:47,110 --> 00:52:52,030
733
00:52:52,030 --> 00:52:56,830
734
00:52:56,830 --> 00:53:01,990
735
00:53:01,990 --> 00:53:06,370
736
00:53:06,370 --> 00:53:11,620
737
00:53:11,620 --> 00:53:15,280
738
00:53:15,280 --> 00:53:19,182
739
00:53:19,182 --> 00:53:24,310
740
00:53:24,310 --> 00:53:28,360
741
00:53:28,360 --> 00:53:32,980
742
00:53:32,980 --> 00:53:38,800
743
00:53:38,800 --> 00:53:44,970
744
00:53:46,505 --> 00:53:52,170
745
00:53:52,170 --> 00:53:56,280
746
00:53:56,280 --> 00:53:59,574
747
00:53:59,574 --> 00:54:04,830
748
00:54:04,830 --> 00:54:08,150
749
00:54:08,150 --> 00:54:11,454
750
00:54:11,454 --> 00:54:15,160
751
00:54:15,160 --> 00:54:20,400
752
00:54:20,400 --> 00:54:24,602
753
00:54:24,602 --> 00:54:29,619
754
00:54:29,619 --> 00:54:34,831
755
00:54:34,831 --> 00:54:38,848
756
00:54:38,848 --> 00:54:43,162
757
00:54:43,162 --> 00:54:46,937
758
00:54:46,937 --> 00:54:50,271
759
00:54:50,271 --> 00:54:54,463
760
00:54:54,463 --> 00:54:58,681
761
00:54:58,681 --> 00:55:02,985
762
00:55:02,985 --> 00:55:07,257
763
00:55:07,257 --> 00:55:10,891
764
00:55:10,891 --> 00:55:17,549
765
00:55:17,549 --> 00:55:24,453
766
00:55:24,453 --> 00:55:31,290
767
00:55:31,290 --> 00:55:37,701
768
00:55:37,701 --> 00:55:43,140
769
00:55:43,140 --> 00:55:46,838
770
00:55:47,361 --> 00:55:51,361
771
00:55:51,361 --> 00:55:54,725
772
00:55:54,725 --> 00:55:58,953
773
00:55:58,953 --> 00:56:04,293
774
00:56:04,293 --> 00:56:09,630
775
00:56:09,630 --> 00:56:13,611
776
00:56:13,611 --> 00:56:17,483
777
00:56:17,483 --> 00:56:21,814
778
00:56:21,814 --> 00:56:26,136
779
00:56:26,136 --> 00:56:31,395
780
00:56:31,395 --> 00:56:35,428
781
00:56:35,428 --> 00:56:39,722
782
00:56:39,722 --> 00:56:44,161
783
00:56:45,811 --> 00:56:50,310
784
00:56:50,310 --> 00:56:54,073
785
00:56:54,073 --> 00:56:57,720
786
00:56:57,720 --> 00:57:02,270
787
00:57:02,270 --> 00:57:06,510
788
00:57:06,510 --> 00:57:10,085
789
00:57:10,085 --> 00:57:12,822
790
00:57:12,822 --> 00:57:16,161
791
00:57:16,161 --> 00:57:19,253
792
00:57:19,253 --> 00:57:22,932
793
00:57:22,932 --> 00:57:26,620
794
00:57:26,620 --> 00:57:30,078
795
00:57:30,078 --> 00:57:33,922
796
00:57:33,922 --> 00:57:38,379
797
00:57:38,379 --> 00:57:43,382
798
00:57:43,382 --> 00:57:46,965
799
00:57:46,965 --> 00:57:51,137
800
00:57:51,137 --> 00:57:55,221
801
00:57:55,221 --> 00:57:58,887
802
00:57:58,887 --> 00:58:02,851
803
00:58:02,851 --> 00:58:06,423
804
00:58:06,423 --> 00:58:09,748
805
00:58:09,748 --> 00:58:12,763
806
00:58:12,763 --> 00:58:17,678
807
00:58:17,678 --> 00:58:22,130
808
00:58:22,130 --> 00:58:26,171
809
00:58:26,171 --> 00:58:29,487
810
00:58:29,487 --> 00:58:33,347
811
00:58:33,347 --> 00:58:37,136
812
00:58:37,136 --> 00:58:41,254
813
00:58:41,254 --> 00:58:45,464
814
00:58:45,464 --> 00:58:49,020
815
00:58:49,020 --> 00:58:53,007
816
00:58:53,007 --> 00:58:55,981
817
00:58:57,221 --> 00:59:01,799
818
00:59:01,799 --> 00:59:04,998
819
00:59:04,998 --> 00:59:10,336
820
00:59:10,336 --> 00:59:13,820
821
00:59:13,820 --> 00:59:17,272
822
00:59:17,272 --> 00:59:21,804
823
00:59:21,804 --> 00:59:26,937
824
00:59:26,937 --> 00:59:31,625
825
00:59:31,625 --> 00:59:36,515
826
00:59:36,515 --> 00:59:42,040
827
00:59:42,040 --> 00:59:48,444
828
00:59:48,444 --> 00:59:54,311
829
00:59:54,311 --> 00:59:59,230
830
00:59:59,230 --> 01:00:01,970
831
01:00:01,970 --> 01:00:05,158
832
01:00:05,158 --> 01:00:10,747
833
01:00:10,747 --> 01:00:15,679
834
01:00:15,679 --> 01:00:19,085
835
01:00:19,085 --> 01:00:22,790
836
01:00:22,790 --> 01:00:25,730
837
01:00:25,730 --> 01:00:29,390
838
01:00:29,390 --> 01:00:33,530
839
01:00:33,530 --> 01:00:37,310
840
01:00:37,310 --> 01:00:41,630
841
01:00:41,630 --> 01:00:45,650
842
01:00:45,650 --> 01:00:49,430
843
01:00:49,430 --> 01:00:54,380
844
01:00:54,380 --> 01:00:57,560
845
01:00:57,560 --> 01:01:02,150
846
01:01:02,150 --> 01:01:08,030
847
01:01:08,030 --> 01:01:11,210
848
01:01:11,210 --> 01:01:16,280
849
01:01:16,280 --> 01:01:20,000
850
01:01:20,000 --> 01:01:24,080
851
01:01:24,080 --> 01:01:27,472
852
01:01:28,688 --> 01:01:33,530
853
01:01:33,530 --> 01:01:36,380
854
01:01:36,380 --> 01:01:39,530
855
01:01:39,530 --> 01:01:42,800
856
01:01:42,800 --> 01:01:45,988
857
01:01:45,988 --> 01:01:48,830
858
01:01:48,830 --> 01:01:52,070
859
01:01:52,070 --> 01:01:57,260
860
01:01:57,260 --> 01:02:01,220
861
01:02:01,220 --> 01:02:05,180
862
01:02:05,180 --> 01:02:08,030
863
01:02:08,030 --> 01:02:11,960
864
01:02:11,960 --> 01:02:17,270
865
01:02:17,270 --> 01:02:20,240
866
01:02:20,240 --> 01:02:23,270
867
01:02:23,270 --> 01:02:27,500
868
01:02:27,500 --> 01:02:32,760
869
01:02:32,760 --> 01:02:37,500
870
01:02:37,500 --> 01:02:41,910
871
01:02:41,910 --> 01:02:46,920
872
01:02:46,920 --> 01:02:52,050
873
01:02:52,050 --> 01:02:55,980
874
01:02:55,980 --> 01:02:59,670
875
01:02:59,670 --> 01:03:03,210
876
01:03:03,210 --> 01:03:06,106
877
01:03:06,818 --> 01:03:11,855
878
01:03:11,855 --> 01:03:40,000