<i>34c3 intro</i>

Herald: ... used Anja Dahlmann, a
political scientist and researcher at

Stiftung Wissenschaft und Politik, a
berlin-based think-tank. Here we go.

<i>applause</i>

Anja Dahlmann: Yeah, Thanks for being
here. I probably neither cut myself nor

proposed but I hope it's still
interesting. I'm going to talk about

preventive arms control and international
humanitarian law and doing in this

international debate around autonomous
weapons. This type of weapon is also

referred to as Lethal Autonomous Weapons
System, short LAWS, or also killer robots.

So if I say LAWS, I mostly mean these
weapons and not like legal laws, just to

confuse you a bit. Okay. I will discuss
this topic along three questions. First of

all, what are we actually talking about
here, what are autonomous weapons? Second,

why should we even care about this? Why's
it important? And third, how could this

issue be addressed on international level?
So. I'll go through my slides, anyway,

what are we talking about here? Well,
during the international negotiations, so

far no real, no common definition has been
found. So States, Parties try to find

something or not and for my presentation I
will just use a very broad definition of

autonomous weapons, which is: Weapons that
can once activated execute a broad range

of tasks or selecting to engage targets
without further human intervention. And

it's just a very broad spectrum of weapons
that might fall under this definition.

Actually, some existing ones are there as
well which you can't see here. That would

be the Phalanx system for example. It's
been around since the 1970s. Sorry...

Herald: Man kann nichts hören
auf der Bühne. Mach mal weiter.

Dahlmann: Sorry. So, Phalanx system has
been around since the 1970s, a US system,

air defense system, based on ships and
it's been to - just yeah, defend the ship

against incoming objects from the air. So
that's around, has been around for quite a

long time and it might be even part of
this LAWS definition or not but just to

give you an impression how broad this
range is: Today, we've got for example

demonstrators like the Taranis drone, a UK
system, or the x74b which can, for

example, autonomously land
<i>applause</i>

land on aircraft carriers and can be air-
refueled and stuff like that which is

apparently quite impressive if you don't
need a human to do that and in the future

there might be even, or there probably
will be even more, autonomous functions,

so navigation, landing, refueling, all
that stuff. That's, you know, old but at

some point there might, be weapons might
be able to choose their own ammunition

according to the situation. They might be
able to choose their target and decide

when to engage with the target without any
human intervention at some point. And

that's quite problematic, I will tell you
why that's in a minute. Overall, you can

see that there's a gradual decline of
human control over weapons systems or over

weapons and the use of force. So that's a
very short and broad impression of what

we're talking about here. And talking
about definitions, it's always interesting

what you're not talking about and that's
why I want to address some misconceptions

in the public debate. First of all, when
we talk about machine autonomy, also

artificial intelligence, with intelligence
which is the technology behind this,

people - not you probably - in the media
and the broader public often get the idea

that these machines might have some kind
of real intelligence or intention or an

entity on own right and they're just not.
It's just statistical methods, it's just

math and you know way more about this than
I do so I will leave it with this and just

say that or highlight that they have these
machines, these weapons have certain

competences for specific tasks. They are
not entities on their own right, they are

not intentional.And that's important when
we talk about ethical and legal challenges

afterwards. Sorry. There it is. And the
other, in connection with this, there's

another one, which is the plethora of
Terminator references in the media as soon

as you talk about autonomous weapons,
mostly referred to as killer robots in

this context. And just in case you tend to
write an article about this: don't use a

Terminator picture, please. Don't, because
it's really unhelpful to understand where

the problems are. With this kind of thing,
people assume that we have problems is

when we have machines with a human-like
intelligence which look like the

Terminator or something like this. And the
problem is that really way before that

they start when you use assisting systems
when you have men or human-machine teaming

or when you accumulate a couple of
autonomous functions through the targeting

cycle. So through this, the military steps
are lead to the use of force or lead to

the killing of people. And that's not,
this is really not our problem at the

moment. So please keep this in mind
because it's not just semantics, semantics

to differentiate between these two things.
It's really manages the expectations of

political and military decision-makers.
Ok, so now you've got kind of an

impression what I'm talking about here so
why should we actually talk about this?

What's all the fuss about? Actually,
autonomous weapons have or would have

quite a few military advantages: They
might be, in some cases, faster or even

more precise than humans. And you don't
need a constant communication link. So you

don't have, you don't have to worry about
instable communication links, you don't

have to worry about latency or detection
or a vulnerability of this specific link.

So yay! And a lot of, let's say very
interesting, military options come from

that. People talk about stealthy
operations and shallow waters for example.

Or you know remote missions and secluded
areas, things like that. And you can get

very creative with tiniest robots and
swarms for example. So shiny new options.

But, and of course there's a "but", it
comes at a prize because you have at least

three dimensions of challenges in this
regard. First of all, the legal ones. When

we talk about these weapons, they might
be, they will be applied in conflict where

international humanitarian law IHL
applies. And IHL consists of quite a few

very abstract principles. For example:
principle of distinction between

combatants and civilians, principle of
proportionality or a military necessity.

They are very abstract and I'm pretty sure
they really always need a human judgment

to interpret this, these principles, and
apply them to dynamic situations. Feel

free to correct me if I'm wrong later. So
that's one thing. So if you remove the

human from the targeting cycle, this human
judgment might be missing and therefore

military decision makers have to evaluate
very carefully the quality of human

control and human judgement within the
targeting cycle. So that's law. Second

dimension of challenges are security
issues. When you look at these new systems

they are cool and shiny and as most new
types of weapons they are, they have the

potential to stir an arms race between
between states. So they actually might

make conflicts more likely just because
they are there and states want to have

them and feel threatened by them. Second
aspect is proliferation. Autonomy is based

on software, so software can be easily
transferred it's really hard to control

and all the other components, or most of
the other components you will need, are

available on the civilian market so you
can build this stuff on your own if you're

smart enough. So we have might have more
conflicts from these types of weapons and

it's might get, well, more difficult to
control the application of this

technology. And the third one which is it
especially worrying for me is the as

potential for escalation within the
conflict, especially when you have, when

both or more sites use these autonomous
weapons, you have these very complex

adversary systems and it will become very
hard to predict how they are going to

interact. They will increase the speed of
the of the conflict and the human might

not even have a chance to process
what's going on there.

So that's really worrying and we can see
for example in high-frequency trading at

the stock markets where problems arise
there and how are difficult is for humans

to understand what's going on there. So
that, that are of some of these security

issues there. And the last and maybe maybe
most important one are ethics. As I

mentioned before, when you use autonomy
and weapons or machines you have

artificial intelligence so you don't have
real intention, a real entity that's

behind this. So the killing decision might
at some point be based on statistical

methods and no one will be involved there
and that's, well, worrying for a lot of

reasons but also it could constitute a
violation of human dignity. You can argue

that humans have, well, you can kill
humans in in war but they at least have

the right to be killed by another human or
at least by the decision of another human,

but we can discuss this later.
So at least on this regard it would be

highly unethical and that really just
scratches the surface of problems and

challenges that would arise from the use
of these autonomous weapons. I haven't

even touched on the problems with training
data, with accountability, with

verification and all that funny stuff
because I only have 20 minutes. So, sounds

pretty bad, doesn't it? So how can this
issue be addressed? Luckily, states have,

thanks to a huge campaign of NGOs, noticed
that there might be some problems and

there might be a necessity to address
that, this issue. They're currently doing

this in the UN Convention on certain
conventional weapons, CCW, where they

discuss a potential ban of the development
and use of these lethal weapons or weapons

that lack meaningful human control over
the use of force. There are several ideas

around there. And such a ban would be
really the maximum goal of the NGOs there

but it becomes increasingly unlikely that
this happens. Most states do not agree

with a complete ban, they want to regulate
it a bit here, a bit there, and they

really can't find a common common
definition as I mentioned before because

if you have a broad definition as just as
I used it you will notice that you have

existing systems in there that might be
not that problematic or that you just

don't want to ben and you might stop
civilian or commercial developments which

you also don't want to do. So states are
stuck on this regard and they also really

challenge the notion that we need a
preventive arms control here, so that we

need to act before these systems are
applied on the battlefield. So at the

moment, this is the fourth year or
something of these negotiations and we

will see how it goes this year and if
states can't find a common ground there it

becomes increasingly like or yeah becomes
likely that it will change to another

forum just like with anti-personnel mines
for example which where the the treaty was

found outside of the United Nations. But
yeah, the window of opportunity really

closes and states and NGOs have to act
there and yeah keep on track there. Just

as a side note, probably quite a few
people are members of NGOs so if you look

at the campaign to stop killer robots with
a big campaign behind this, this process,

there's only one German NGO which is
facing finance, so if you're especially if

you're German NGO and are interest that in
AI it might be worthwhile to look into the

military dimension as well. We really need
some expertise on that regard, especially

on AI and these technologies. They're...
Okay, so just in case you fell asleep in

the last 15 minutes I want you to take
away three key messages: Please be aware

of the trends and internal logic that lead
to autonomy in weapons. Do not

overestimate the abilities of autonomy, of
autonomous machines like intent and these

things and because you probably all knew
this already, please tell people about

this, tell other people about this,
educate them about this type of

technology. And third, don't underestimate
the potential dangers for security and

human dignity that comes from this type of
weapon. I hope that I could interest you a

bit more in this in this particular issue
if you want to learn more you can find

really interesting sources on the website
of the CCW at the campaign to stuff killer

robots and from a research project that I
happen to work in, the International Panel

on the Regulation of Autonomous Weapons,
we do have a few studies on that regard

and we're going to publish a few more. So
please, check this out and thank you for

your attention.

<i>Applause</i>

Questions?

Herald: Sorry. So we have some time for
questions answers now. Okay, first of all

I have to apologize that we had a hiccup
with the signing language, the acoustics

over here on the stage was so bad that she
didn't could do her job so I'm

terrible sorry about that. We fixed it in
the talk and my apologies for that. We are

queuing here on the microphones already so
we start with microphone number one, your

question please.
Mic 1: Thanks for your talk Anja. Don't

you think there is a possibility to reduce
war crimes as well by taking away the

decision from humans and by having
algorithms who decide which are actually

auditable?
Dahlmann: Yeah that's, actually, that's

something I just discussed in the
international debate as well, that there

might, that machines might be more ethical
than humans could be. And well, of course

they won't just start raping women because
they want to but you can program them to

do this. So you just you shift the
problems really. And also maybe these

machines don't get angry but they don't
show compassion either so if you are there

and your potential target they just won't
stop they will just kill you and do not

think once think about this. So you have
to really look at both sides there I guess.

Herald: Thanks. So we switch over
to microphone 3, please.

Mic 3: Thanks for the talk. Regarding
autonomous cars, self-driving cars,

there's a similar discussion going on
regarding the ethics. How should a car

react in a case of an accident? Should it
protect people outside people, inside,

what are the laws? So there is another
discussion there. Do you work with people

in this area or is this is there any
collaboration?

Dahlmann: Maybe there's less collaboration
than one might think there is. I think

there is. Of course, we we monitor this
debate as well and yeah we think about the

possible applications of the outcomes for
example from this German ethical

commission on self-driving cars for our
work. But I'm a bit torn there because

when you talk about weapons, they are
designed to kill people and cars mostly

are not. So with this ethical committee
you want to avoid killing people or decide

what happens when this accident occurs. So
they are a bit different but of course

yeah you can learn a lot from both
discussions and we aware of that.

Herald: Thanks. Then we're gonna go over
in the back, microphone number 2, please.

Mic 2: Also from me thanks again for this
talk and infusing all this professionalism

into the debate because some of the
surroundings of our, so to say ours

scenery, they like to protest against very
specific things like for example the

Rammstein air base and in my view that's a
bit misguided if you just go out and

protest in a populistic way without
involving these points of expertise that

you offer. And so, thanks again for that.
And then my question: How would you

propose that protests progress and develop
themselves to a higher level to be on the

one hand more effective and on the other
hand more considerate of what is at stake

on all the levels and on
all sides involved?

Dahlmann: Yeah well, first, the Rammstein
issue is completely, actually a completely

different topic. It's drone warfare,
remotely piloted drones, so there are a

lot of a lot of problems with this and
we're starting killings but it's not about

lethal autonomous weapons in particular.
Well if you want to be a part of this

international debate, there's of course
this campaign to stop killer robots and

they have a lot of really good people and
a lot of resources, sources, literature

and things like that to really educate
yourself what's going on there, so that

would be a starting point. And then yeah
just keep talking to scientists about

this and find out where we see the
problems and I mean it's always helpful

for scientists to to talk to people in the
field, so to say. So yeah, keep talking.

Herald: Thanks for that. And the
signal angel signaled that we have

something from the internet.
Signal Angel: Thank you. Question from

IRC: Aren't we already in a killer robot
world? The bot net can attack a nuclear

power plant for example. What do you think?
Dahlmann: I really didn't understand a

word, I'm sorry.
Herald: I didn't understand that as well,

so can you speak closer to
the microphone, please?

Signal Angel: Yes. Aren't we already in a
killer robot world?

Herald: Sorry, that doesn't work. Sorry.
Sorry, we stop that here, we can't hear it

over here. Sorry.
Signal Angel: Okay.

Herald: We're gonna switch over to
microphone two now, please.

Mic 2: I have one little question. So in
your talk, you were focusing on the

ethical questions related to lethal
weapons. Are you aware of ongoing

discussions regarding the ethical aspects
of the design and implementation of less

than lethal autonomous weapons for crowd
control and similar purposes?

Dahlmann: Yeah actually within the CCW,
every term of this Lethal Autonomous

Weapon Systems is disputed also the
"lethal" aspect and for the regulation

that might be easier to focus on this for
now because less than lethal weapons come

with their own problems and the question
if they are ethical and if they can, if

IHL applies to them but I'm not really
deep into this discussion. So I'll just

have to leave it there.
Herald: Thanks and back here to microphone

one, please.
Mic 1: Hi. Thank you for the talk very

much. My question is in the context of the
decreasing cost of both, the hardware and

software, over the next say 20, 40 years.
Outside of a nation-state context like

private forces or non nation-state actors
gaining use of these weapons, do things

like the UN convention or the campaign to
stop killer robots apply are they

considering private individuals trying to
leverage these against others?

Dahlmann: Not sure what the campaign says
about this, I'm not a member there. The

the CCW mostly focuses on international
humanitarian law which is important but I

think it's it's not broad enough. So
questions like proliferation and all this

is connected to your question and not
really or probably won't be part of

regulation there. It's discussed only on
the edges of the of the debates and

negotiations there but it doesn't seem to
be a really issue there.

Mic 1: Thanks.
Herald: And over to microphone six,

please.
Mic 6: Thank you. I have a question as a

researcher: Do you know how far the
development has gone already? So how

transparent or intransparent is your look
into what is being developed and

researched on the side of militaria
working, military people working with

autonomous weapons and developing them?
Dahlmann: Well, for me it's quite

intransparent because I only have only
access to public publicly available

sources so I don't really know what what's
going on behind closed doors in the

military or in the industry there. Of
course you can you can monitor the

civilian applications or developments
which can tell a lot about the the state

of the art and for example the DARPA
the American Development Agency, they

published sometimes a call for papers,
that's not the term, but there you can see

where in which areas they are interested
in then for example they really like this

idea of autonomous killer bug that can
act in swarms and monitor or even kill

people and things like that. So yeah we
try to piece it, piece it together in

our work.
Herald: We do have a little bit more time,

are you okay to answer more questions?
Dahlmann: Sure.

Herald: Then we're gonna switch over to
microphone three, please.

Mic 3: Yes, hello. I think we are living
already in a world of Leathal Autonomous

Weapon Systems if you think about these
millions of landmines which are operating.

And so the question is: Shouldn't it be
possible to ban these weapon systems the

same way as land mines that are already
banned by several countries so just

include them in that definition? And
because the arguments should be very

similar.
Dahlmann: Yeah it does, it does come to

mind of course because these mines are
just lying around there and no one's

interacting when you step on them and
boom! But they are, well it depends, it

depends first of all a bit of your
definition of autonomy. So some say

autonomous is when you act in dynamic
situations and the other ones would be

automated and things like that and I think
this autonomy aspect, I really don't want

to find, don't want to find define
autonomy here really but this this action

in more dynamic spaces and the aspect of
machine learning and all these things,

they are way more complex and they bring
different problems than just land mines.

Landmines are problematic, anti-personnel
mines are banned for good reasons but they

don't have the same problems I think. So
it won't be, I don't think it won't be

sufficient to just put the LAWS in there,
the Lethal Autonomous Weapons.

Herald: Thank you very much. I can't see
anyone else queuing up so therefore, Anja,

thank you very much it's your applause!

<i>applause</i>

and once again my apologies that
that didn't work

<i>34c3 outro</i>

subtitles created by c3subtitles.de
in the year 2018. Join, and help us!