What's It Doing Now? (33c3)

[Music]

MC: so have you ever called IT only to be
told

have you tried turning it off and on again?

Today we will be talking about
pilots usually pilots are called to pilot

in command and we expect a pilot to be
just that in command but today's pilots

are turning more and more into computer
operators and have less and less actual

hands-on flying ability so now imagine you
are the pilot flying a gigantic computer

in 30,000 feet height with 200 souls
behind you only to be told by IT have you

tried turning it off and on again so I
would like to welcome Bernd Sieker. who is

a systems engineer and an aviation
accident analyst he specialized in reverse

engineering and he's developing formal
methods to development of safety critical

systems and he will enlighten us about
problems in aviation automation because

apparently every pilot has uttered the
words what's it doing now

[Applause]
BS: yes thank you thank you very much yes

but first I'd like to learn a bit about
the audience so how many of you here in

the hall today are pilots Oh quite a few
so commercial pilots? far fewer. ATP

anyone? yeah there's one I heard one but I
can't see. okay so some of you will know

about some of the stuff I hope there's a
bit new stuff for everyone let's get right

into it what the announcer said was a bit
of nice folklore it's not completely true

but there's a little bit truth to it what
I'm going to talk about is automation in

the aircraft and the idea is often as he
said that it's just a computer and the

pilot doesn't have to do anything that's
one saying that in modern airplanes there

will only be one pilot and a dog and the
pilot is there to watch what the the

pilots there to feed the dog and the dog
is there to bite the pilot if he touches

anything so that's not quite yet how it is
I talk a very little bit about the

analysis method that we use to analyze
accidents not only in aviation but mostly

and then I'll tell you a short tale of two
throttles or two thrust levers as they are

sometimes called and also talk about human
pilots how they cope with failures or

don't as the case may be and I haven't
seen a lot of other talks here about self-

driving cars although they are now
becoming a very big thing so I'll touch

that briefly and have a tentative
conclusion. I can't see very fine to the

future so I'm not sure if I'm right about
that. So what is automation in airplanes?

the most obvious thing is that automated
flight controls on every airliner and on

many small airplanes these days. there
used to be a requirement for a simple

autopilot even on small private single-
engine airplanes if you want to fly under

instrument flight rules. That has been
relaxed somewhat now but many small planes

still have them. so there are three levels
of fat controls there the first one is

manual flight where the pilot moves flight
controls and the airplane does what it's

told then there's the simple
autopilot where the pilot just

sets airspeed altitude climb rate stuff
like that and there are managed modes now

where there's a more sophisticated
computer which has knowledge about the

whole flight with waypoints and altitudes
and there are other automated systems not

only the flight controls spoilers on the
ground have to extend to help slowing down

the aircraft the high-lift devices are
automated radios maybe Auto-tuning there's

the computer that controls the engines
full authority digital engine control.

There are things like cabin pressurization
and many other small subsystems are

automated as they are in cars these days.
So what is automation not? It's not yet

except for very few specialized drones a
self flying aircraft the pilot in command

still is in command at all times you can
turn off the automation you can have to

fly the aircraft at any time if you wants
to and barring any serious errors which

are extremely rare in commercial aircraft
the airplane does what it's told. the

pilots the the autopilot really doesn't
have any decision capabilities except at

the very lowest level deciding on a bank
angle to make the right turn and things

like that it is also not a panacea for any
errors that the pilot can make now you can

still fly a highly computerized modern
aircraft into the side of the mountain if

you want to. so some military aircraft
actually have systems that will prevent

you from flying into a mountain if they're
active or if you have passed out but

airliners don't at the time and of course
the pilot in command still bears the

ultimate responsibility for the safe
conduct of the flight. so as I said

briefly manual flying is just stick and
rudder you move the stick and you move

your rudder pedals and the airplane moves
the control surfaces mechanically on small

airplanes hydrolically assisted or even
computer-assisted on some airliners on

most modern airliners called fly-by-wire
you may have heard about that then the

simple autopilot modes where you
directly select the heading and the

airplane flies in that heading and managed
modes as I said before where you have a

sophisticated flight management system
which then in turn sets headings and climb

rates and things like that on the
autopilot proper. they are not super

reliable they can be thrown off by many
things and mostly they they turn off when

there's any small error in any of the
small subsystems, any of the various input

values that you get air speed altitude
engine power anything if any of those have

invalid readings it'll turn off and the
pilots have to assume command in that

case. they cannot handle basically
anything unexpected most air sensors are

there then threefold so if only one of
them disagrees the other two are usually

taken as valid but if they all three
disagree then the system just says I don't

know what's true anymore what speed is and
all the automatics drop out and most of

the computer assisted manual flying also
is turned off in that case.

so this is very briefly the method

that we have developed at
the University of Bielefeld

and the professor lepkin for analyzing
accidents called why because analysis it

uses a formal notion of causality called
the counterfactual test and then you can

make a very nice graph for accidents
they're usually bigger than that but it's

more or less objective criterion for
causality and then different people with

some experience in the domain make why
because graphs of an accident they usually

are very similar to each other

so there's a lot of automation
on modern airplanes

and it's quite hard to get it right and
one of the reasons is that unlike for many

situations in cars and rail vehicles there
is no default safe state, you can't just

turn everything off and stop by the
roadside so we always have to decide the

engineers always have to plan for many
eventualities what can happen in the air

and decide what given a certain set of
circumstances is the safest state for the

airplane to be in and that is not always
unambiguous and it's a very hard decision

tomake and sometimes
they get it wrong and

sometimes you just get into that situation
where in most cases the set of values the

set of measured values that the system
gets when most circumstance is one set of

decisions is the correct one and you get
into that situation where the computers

get the same inputs and that decision is
the wrong one and that may still lead to

an accident. those are very few and very
rare but these these things can happen. so

a few of the decisions that the engineers
have to take when designing the automation

in airplanes is what to do if things fail
if certain individual things fail if a

combination of things fail little motors
little engines sensors fail some actuators

fail a hydraulic system fails anything
like that what you do in that case with

the remaining systems and what to tell the
pilots? well naively you might assume the

pilot wants to know about everything that
is broken every little valve every little

system that is broken on the airplane but
if a lot goes wrong at the same time then

the decision has to be taken which of
these things that have gone wrong are the

most important for the flight crew to know
and that's not trivial at all and it can

very easily lead to to sensory saturation
of the pilots so they don't know what is

what anymore because from all sides alarms
are blaring there are lots and lots of

displays that they have to watch and so
certain error messages are suppressed in

certain states of flight certain stages of
the flight so as not to overwhelm the

pilot. and some things that may be
essential to have on the ground some

functions for example the wing spoilers
those are the big the big flaps on the top

of the wings that come up after touchdown
are important to have on landing to dump

the lift so the airplane doesn't jump up
again. because it is a touchdown still at

the speed at which it could fly at least
for airliners, for small airplanes it's a

bit different but airliners are safely
above the very lowest

speed they can go when they touchdown so
they need to have some means to make sure

they don't jump up again they still do
sometimes but not very often but the

spoilers destroy most of the lift so
deploying them in the air close to the

ground is extremely dangerous so the
computer has to be absolutely certain so

to speak to know that the aircraft is on
the ground when it gives the command to

deploy the ground spoilers if it does that
a few seconds too early when the airplane

is still a hundred meters up above the
ground that will likely be a fatal accident.

so in most at least in most jet
airliners not in all propeller-driven but

in all almost all jet airliners there's an
automatic thrust management so the

computer does not only control where the
nose of the airplane points but also how

much power the engines produce and there
are two different one might call them

philosophies between the two major air
framers and Boeing and most others to use

back driven throttles so the computer sets
the thrust and moves the thrust levers to

match the commanded thrust position and
Airbus has a different system where the

thrust levers remain in one position
throughout the entire flight basically

after take off when thrust is reduced
for the main climb and cruise and descend

and everything they remain in one position
and the computer tells the engines

directly which thrust to produce. and
there are there's an argument which one of

the systems is better but I'll show you
accidents three accidents in which the

thrust system the throttle system played a
role. so the first one has a little video

you will see I think there are two
different camera perspectives you will see

two airplanes landing of the same class
they are small airliners two hundred

people something like that 150 to 200 now
landing and the first one is a normal landing

so it's already pretty slow takes
its time

and the next one is the accident flight.

it's on the same day it's only
minutes apart so on the same Airport

and you can see that one is slowed down
and the one other one is still going very fast

so there's the first one

and that's the second one

and as you can imagine that didn't end well

it was one of the worst aviation
accidents maybe still

the worst today in Brazil where 200 people
died

and as you can see this is a

transcript of of the flight data recorder
the digital flight data recorder and the

first two lines are the interesting ones
that says TLA that is thrust lever angle

and normally what happens on landing just
before touchdown the pilot pulls both

thrust wheels to idle now the engine
thrust goes down to to idle and then it

touches down engages reverse thrust
spoilers brakes everything to slow down

and what happened in this case is that the
pilot only moved one of the thrust to idle

and left the other there put the one
thrust lever in reverse but not the other

and that led to the computer getting
conflicting information about whether the

pilots actually wanted to land or not so
it didn't deploy the automatic wheel

brakes it didn't deploy the spoilers and
reverse thrust only on one engine so that

went pretty badly and some people said
well with tactile feedback from a thrust

levers if the pilots have been used to
that they would have noticed earlier and

we can't really be sure because the pilots
also died in the accident but there were

some people who made a case that moving
thrust levers would have been a lot better

in this case

so is that always better?

here's another throttle related accident

in this time it was a Boeing Boeing 737 at
Amsterdam Schiphol Airport there was a

small technical malfunction would call
which caused the computers to think the

airplane was actually eight feet
underground that was the reading that it

gave due to work through the way it works
and so said oh I'm below 30 feet I have to

reduce the thrust to idle and that's what
it did although it was still a couple

hundred feet high and the pilots didn't
notice early enough and let the

speed decay and the wing stalled and
crashed the airplane crashed and the nine

people died it was moderately only a
moderately hard crash so most people

survived actually though it was still a
problem and the way the auto in auto

throttle system works in this case if the
thrust levers had been static this

wouldn't have happened because the pilots
were pushed the thrust levers above a

certain detent and it wouldn't have
reduced thrust automatically again so it's

very hard to say which system in total is
better you can count the accidents maybe

in which it played a roll but there are so
few they're just really less than a

handful in each case so they're not
statistically significant so you can't

really say by statistics alone which
system is better than the other they both

have their own problems and this is one of
the decisions as engineers that you really

can't make a decisive argument for so one
manufacturer chooses one and the other

chooses the other and there's another one
is asiana flight 214 at San Francisco many

of you may remember that. only three
people were killed in this one because it

really burned out only after the crash
after everyone had evacuated and so the

auto throttles didn't work as expected in
this case the pilots thought oh the auto

throttles will hold the speed we don't
have to worry about that as far as I

remember there were five pilots in the
cockpit and when finally someone noticed

and pushed the throttles forward it was
already too late the engines take that

time to spool up the legal requirement is
that they may take up to eight seconds to

spool up from idle to the necessary power
to go around and there wasn't enough time

for that because after the engines have
spooled up the airplane also still has to

accelerate to get back to flying speed
again so in this case again the wings

stalled the airplane crashed just short of
the runway and three people died. and the

third case was even one when nothing was
wrong with the airplane except you could

argue it was a design flaw but it was
working as designed people who were going

to fly the aircraft learned how the system
worked learned everything about it

hopefully and so more training may perhaps
be the answer that is one thing system

knowledge, two. crew resource management
has been a big thing in previous decades

that the pilot command in command is not a
dictator on the airplane he has to listen

to the others to the other pilot even though
he has ultimate authority in decision.

so do pilots always screw up if
the automation fails? no luckily not if

other systems fail in this case not the
automation really but there are two

cases which I would briefly mentioned
Chesley Sullenberger everybody knows about

him the movie has just been out the
ditching in the Hudson superb pilot great

decision making to find the biggest flat
surface in the area to pull it down and

Peter Burkhill he'll many other so who
knew about Peter Burkhill? A few. he was

the one saved about as many people as
Sullenberger when on approach to london

heathrow both engines lost thrust most of
the thrust anyway and he managed to put it

down within the airport but short of the
runway it was a crash landing the airplane

was destroyed but nobody died so it was a
pretty good outcome.

so airplanes are one thing,
another thing are cars

and anyone here has a
self-driving car? Or at least a

lane assist or something? Not many so not
many people don't trust these newfangled

systems I guess. one of the big
differences is that pilots who are going

to fly highly automated aircraft have to
take a long training course beyond their

pilot's license to learn the specifics of
operating this specific aircraft and

maintenance is very highly controlled and
regulated so that's another thing. and the

things for cars in general if something's
wrong with the engine you can just pull

over to the right and stop in most cases
and cars cannot just take off and take

evasive action in the third dimension and

there are lots and lots of obstacles
on the ground there are trees

cars people houses everything whereas
the air is mostly empty not entirely

air-to-air collisions happen midair collisions do
happen but they are very very few. in the

automatic systems in the self-driving cars
or the autonomous cars that we have today

require constant monitoring and if the
systems work too well then drivers may

actually forget about it and think they
are perfect and let their attention

wander. pilots sometimes are prone to do
that as well but the thing is that in

Cruise in cruise flight if the automatics
drop out the pilots have on the order of

minutes to react really at least several
seconds whereas on a road car if the

automatics drop out and you're in a curve
you have fractions of a second to save the

car with the current state of the
technology.

some of you probably have heard about the trolley
problem or trolley-ology as it's sometimes called.

it basically boils down to that a
fully autonomous car a highly

automated car may eventually have to make
the decision between killing the occupants

and killing people on the road. and I
think that is fundamentally an unsolvable

ethical problem that we cannot just leave
to the engineers or the car manufacturers

to decide that maybe the occupants are
always more important than people on the

road? what if there's only one person in
the car and there's a crowd on the road

and you have to decide between steering on
the car into to the tree and killing the

sole occupant or killing several people
that are in front of the car these are

situations that may actually happen. so I
really can't see what the right answer is

to that if there is one and maybe there
isn't one. some engineers have actually

suggested that making a random decision in
that case is the answer. I'm not too sure

about that either but whatever the
decision the software takes at that moment

then people will die and they will take
the blame either way and we don't know yet

how that's going to turn out in front of
the courts.

so automation is hard to get

right and in some cases self-driving cars
it may be impossible to get it absolutely

right. which stage is the safest for the
systems to be in and at what time who

knows it's very very hard to get it right
even in limited systems such as airplanes

and what to display to the operators and
when in many cases it would help the

pilots a lot when the automation drops out
to know intimate details of how the system

works internally. airbus has some logic
diagrams in their pilots handbook but they

are labeled 'for info' which means they
are not required for any exams it's just

interesting to know but in case of the
logic for extension of the ground spoilers

it's quite helpful to know which
conditions exactly have to be satisfied

for the ground spoilers to deploy.

but some of these problems I think cannot

be left to engineers and scientists alone
and we need psychologists and maybe

sociologists other people who know about
the psyche of people who know about how

people think how people react how people
process information to make good

engineering design decisions to build
safer systems. and as I said some of the

fundamental ethical problems may turn out
to remain unsolvable.

thank you I think we have a little bit of
time for questions

[Applause]

MC: yes we actually do we have some time

for questions and we're gonna start with
the internet if there are any questions no

there are not then it's microphone number
three

Q: yes you mentioned the ethical problem
of the decision making the trolley problem

so whenever this comes up regarding
automated driving systems whether it be

flight control or car driving I always get
a little bit mad when philosophers come up

with that the there is one decisive
decision you can make and that is the

whole thing should act predictably
especially in road traffic the uttermost

importance is that all participants behave
predictably swerving out of lane is the

most dangerous thing you can do
H: And what's your question?

Q: and if you have to make
this decision people say you

have to make a decision then I say no
there is a definitive safe state that is

drive with enough distance to the guy in
front of you don't tailgate don't speed up

because if you're a regular driver
MC: no no please ask your question

Q: okay question is why are people always
saying it's ethically not decisive

decidable?
BS: it isn't because if if just keeping

enough distance we solve all problems that
would be that would be fine but cars are

not the only participants in traffic there
are people right and they can just jump in

front of a car. that is not predictable,
yeah you can you can require people to

behave predictable but good luck with
that.

Q: I would like to counter that
[Applause]

MC: okay I'm sorry there's not much room
for discussion right now but microphone

number two please ask a concise question
Q: okay let me try so you said about

automation in airplanes that whenever
there is a small malfunction the autopilot

will disconnect and expect the pilots to
fix the situation right so it is my-

BS: yeah it's not not the smallest problem
but some yeah

Q: okay so but it is my understanding that
the pilots are still expected to follow

procedures and not make any random gut
decisions in most cases. my question is do

you have statistics when the standard
procedures were actually not applicable in

how many cases and in how many of these
cases did the pilots actually managed to

save the flight?
BS: no I'm not aware of any statistics and

one of the problems with that is that in
general the data recorder is only read

when there was an accident and it is
strictly off-limits in all other

circumstances some airplanes have a quick
access data recorder which they can

routinely read but only anonymized so and
I don't think the airplanes published

statistics about that
MC: okay last question microphone number

four please
Q: yeah I'm just I just want to bring this

back to this sort of the IT security part
where what I find very good about about

the way accidents are handled in in
aviation is that the report is completely

public so if you want to read you know the
Challenger cut this show if you can you

can actually read all the technical
details and all that all the stuff that

happened and and all that information is
there and the question is is why is this

not happening in the IT sector where
clearly millions of people are being

affected and somehow you haven't reached
this stage where the the the data and the

analysis of the data is public so we can
all learn from it and get better as it has

been you know the way
BS: I think the short answer is excuse me

Q: no it's good
BS: I think the short answer is because

there is no legal requirement and if there
weren't for accident reports to be

distributed then many airlines wouldn't do
it .

Q: but why? It's very clear [???]
BS: it's because it's embarrassing if you

have an accident it's basically the thing
I think

MC: ok very last question microphone
number one please

Q: hey so one of the reasons we have
automation in aircraft in the first place

is to reduce pilot workload where too high
pilot workload is a major cause of

accidents it seems like one of the issues
we're talking about here is that in a

situation where something's gone wrong the
presence of that automation are needed to

understand it means you've got a higher
pilot workload in that situation the

question what is it doing now what's the
industry sort of approach to that effect

and what do you think about that?
BS: I think the traditional approach is to

just pile on more automation so then if
that fails the pilot has an even higher

workload but the current thing is that
manufacturers and the airlines go back

very very slowly to letting the pilot hand
fly more often and for a long time the the

mantra was use automation whenever
possible the highest level of automation

that is appropriate for the situation so
only the takeoff and touchdown were flown

by hand and now it is very often use the
appropriate level of automation and that

means if there's not very high workload
and not a lot of traffic then hand fly the

approach for example. so to to keep in
good practice and right to maintain

proficiency for all situations hopefully.

MC: thank you and please give a warm hand
of applause for Bernd Sieker.

[Applause]

[Music]

subtitles created by c3subtitles.de
in the year 2018. Join, and help us!