34c3 intro
Herald: ... used Anja Dahlmann, a
political scientist and researcher at
Stiftung Wissenschaft und Politik, a
berlin-based think-tank. Here we go.
applause
Anja Dahlmann: Yeah, Thanks for being
here. I probably neither cut myself nor
proposed but I hope it's still
interesting. I'm going to talk about
preventive arms control and international
humanitarian law and doing in this
international debate around autonomous
weapons. This type of weapon is also
referred to as Lethal Autonomous Weapons
System, short LAWS, or also killer robots.
So if I say LAWS, I mostly mean these
weapons and not like legal laws, just to
confuse you a bit. Okay. I will discuss
this topic along three questions. First of
all, what are we actually talking about
here, what are autonomous weapons? Second,
why should we even care about this? Why's
it important? And third, how could this
issue be addressed on international level?
So. I'll go through my slides, anyway,
what are we talking about here? Well,
during the international negotiations, so
far no real, no common definition has been
found. So States, Parties try to find
something or not and for my presentation I
will just use a very broad definition of
autonomous weapons, which is: Weapons that
can once activated execute a broad range
of tasks or selecting to engage targets
without further human intervention. And
it's just a very broad spectrum of weapons
that might fall under this definition.
Actually, some existing ones are there as
well which you can't see here. That would
be the Phalanx system for example. It's
been around since the 1970s. Sorry...
Herald: Man kann nichts hören
auf der Bühne. Mach mal weiter.
Dahlmann: Sorry. So, Phalanx system has
been around since the 1970s, a US system,
air defense system, based on ships and
it's been to - just yeah, defend the ship
against incoming objects from the air. So
that's around, has been around for quite a
long time and it might be even part of
this LAWS definition or not but just to
give you an impression how broad this
range is: Today, we've got for example
demonstrators like the Taranis drone, a UK
system, or the x74b which can, for
example, autonomously land
applause
land on aircraft carriers and can be air-
refueled and stuff like that which is
apparently quite impressive if you don't
need a human to do that and in the future
there might be even, or there probably
will be even more, autonomous functions,
so navigation, landing, refueling, all
that stuff. That's, you know, old but at
some point there might, be weapons might
be able to choose their own ammunition
according to the situation. They might be
able to choose their target and decide
when to engage with the target without any
human intervention at some point. And
that's quite problematic, I will tell you
why that's in a minute. Overall, you can
see that there's a gradual decline of
human control over weapons systems or over
weapons and the use of force. So that's a
very short and broad impression of what
we're talking about here. And talking
about definitions, it's always interesting
what you're not talking about and that's
why I want to address some misconceptions
in the public debate. First of all, when
we talk about machine autonomy, also
artificial intelligence, with intelligence
which is the technology behind this,
people - not you probably - in the media
and the broader public often get the idea
that these machines might have some kind
of real intelligence or intention or an
entity on own right and they're just not.
It's just statistical methods, it's just
math and you know way more about this than
I do so I will leave it with this and just
say that or highlight that they have these
machines, these weapons have certain
competences for specific tasks. They are
not entities on their own right, they are
not intentional.And that's important when
we talk about ethical and legal challenges
afterwards. Sorry. There it is. And the
other, in connection with this, there's
another one, which is the plethora of
Terminator references in the media as soon
as you talk about autonomous weapons,
mostly referred to as killer robots in
this context. And just in case you tend to
write an article about this: don't use a
Terminator picture, please. Don't, because
it's really unhelpful to understand where
the problems are. With this kind of thing,
people assume that we have problems is
when we have machines with a human-like
intelligence which look like the
Terminator or something like this. And the
problem is that really way before that
they start when you use assisting systems
when you have men or human-machine teaming
or when you accumulate a couple of
autonomous functions through the targeting
cycle. So through this, the military steps
are lead to the use of force or lead to
the killing of people. And that's not,
this is really not our problem at the
moment. So please keep this in mind
because it's not just semantics, semantics
to differentiate between these two things.
It's really manages the expectations of
political and military decision-makers.
Ok, so now you've got kind of an
impression what I'm talking about here so
why should we actually talk about this?
What's all the fuss about? Actually,
autonomous weapons have or would have
quite a few military advantages: They
might be, in some cases, faster or even
more precise than humans. And you don't
need a constant communication link. So you
don't have, you don't have to worry about
instable communication links, you don't
have to worry about latency or detection
or a vulnerability of this specific link.
So yay! And a lot of, let's say very
interesting, military options come from
that. People talk about stealthy
operations and shallow waters for example.
Or you know remote missions and secluded
areas, things like that. And you can get
very creative with tiniest robots and
swarms for example. So shiny new options.
But, and of course there's a "but", it
comes at a prize because you have at least
three dimensions of challenges in this
regard. First of all, the legal ones. When
we talk about these weapons, they might
be, they will be applied in conflict where
international humanitarian law IHL
applies. And IHL consists of quite a few
very abstract principles. For example:
principle of distinction between
combatants and civilians, principle of
proportionality or a military necessity.
They are very abstract and I'm pretty sure
they really always need a human judgment
to interpret this, these principles, and
apply them to dynamic situations. Feel
free to correct me if I'm wrong later. So
that's one thing. So if you remove the
human from the targeting cycle, this human
judgment might be missing and therefore
military decision makers have to evaluate
very carefully the quality of human
control and human judgement within the
targeting cycle. So that's law. Second
dimension of challenges are security
issues. When you look at these new systems
they are cool and shiny and as most new
types of weapons they are, they have the
potential to stir an arms race between
between states. So they actually might
make conflicts more likely just because
they are there and states want to have
them and feel threatened by them. Second
aspect is proliferation. Autonomy is based
on software, so software can be easily
transferred it's really hard to control
and all the other components, or most of
the other components you will need, are
available on the civilian market so you
can build this stuff on your own if you're
smart enough. So we have might have more
conflicts from these types of weapons and
it's might get, well, more difficult to
control the application of this
technology. And the third one which is it
especially worrying for me is the as
potential for escalation within the
conflict, especially when you have, when
both or more sites use these autonomous
weapons, you have these very complex
adversary systems and it will become very
hard to predict how they are going to
interact. They will increase the speed of
the of the conflict and the human might
not even have a chance to process
what's going on there.
So that's really worrying and we can see
for example in high-frequency trading at
the stock markets where problems arise
there and how are difficult is for humans
to understand what's going on there. So
that, that are of some of these security
issues there. And the last and maybe maybe
most important one are ethics. As I
mentioned before, when you use autonomy
and weapons or machines you have
artificial intelligence so you don't have
real intention, a real entity that's
behind this. So the killing decision might
at some point be based on statistical
methods and no one will be involved there
and that's, well, worrying for a lot of
reasons but also it could constitute a
violation of human dignity. You can argue
that humans have, well, you can kill
humans in in war but they at least have
the right to be killed by another human or
at least by the decision of another human,
but we can discuss this later.
So at least on this regard it would be
highly unethical and that really just
scratches the surface of problems and
challenges that would arise from the use
of these autonomous weapons. I haven't
even touched on the problems with training
data, with accountability, with
verification and all that funny stuff
because I only have 20 minutes. So, sounds
pretty bad, doesn't it? So how can this
issue be addressed? Luckily, states have,
thanks to a huge campaign of NGOs, noticed
that there might be some problems and
there might be a necessity to address
that, this issue. They're currently doing
this in the UN Convention on certain
conventional weapons, CCW, where they
discuss a potential ban of the development
and use of these lethal weapons or weapons
that lack meaningful human control over
the use of force. There are several ideas
around there. And such a ban would be
really the maximum goal of the NGOs there
but it becomes increasingly unlikely that
this happens. Most states do not agree
with a complete ban, they want to regulate
it a bit here, a bit there, and they
really can't find a common common
definition as I mentioned before because
if you have a broad definition as just as
I used it you will notice that you have
existing systems in there that might be
not that problematic or that you just
don't want to ben and you might stop
civilian or commercial developments which
you also don't want to do. So states are
stuck on this regard and they also really
challenge the notion that we need a
preventive arms control here, so that we
need to act before these systems are
applied on the battlefield. So at the
moment, this is the fourth year or
something of these negotiations and we
will see how it goes this year and if
states can't find a common ground there it
becomes increasingly like or yeah becomes
likely that it will change to another
forum just like with anti-personnel mines
for example which where the the treaty was
found outside of the United Nations. But
yeah, the window of opportunity really
closes and states and NGOs have to act
there and yeah keep on track there. Just
as a side note, probably quite a few
people are members of NGOs so if you look
at the campaign to stop killer robots with
a big campaign behind this, this process,
there's only one German NGO which is
facing finance, so if you're especially if
you're German NGO and are interest that in
AI it might be worthwhile to look into the
military dimension as well. We really need
some expertise on that regard, especially
on AI and these technologies. They're...
Okay, so just in case you fell asleep in
the last 15 minutes I want you to take
away three key messages: Please be aware
of the trends and internal logic that lead
to autonomy in weapons. Do not
overestimate the abilities of autonomy, of
autonomous machines like intent and these
things and because you probably all knew
this already, please tell people about
this, tell other people about this,
educate them about this type of
technology. And third, don't underestimate
the potential dangers for security and
human dignity that comes from this type of
weapon. I hope that I could interest you a
bit more in this in this particular issue
if you want to learn more you can find
really interesting sources on the website
of the CCW at the campaign to stuff killer
robots and from a research project that I
happen to work in, the International Panel
on the Regulation of Autonomous Weapons,
we do have a few studies on that regard
and we're going to publish a few more. So
please, check this out and thank you for
your attention.
Applause
Questions?
Herald: Sorry. So we have some time for
questions answers now. Okay, first of all
I have to apologize that we had a hiccup
with the signing language, the acoustics
over here on the stage was so bad that she
didn't could do her job so I'm
terrible sorry about that. We fixed it in
the talk and my apologies for that. We are
queuing here on the microphones already so
we start with microphone number one, your
question please.
Mic 1: Thanks for your talk Anja. Don't
you think there is a possibility to reduce
war crimes as well by taking away the
decision from humans and by having
algorithms who decide which are actually
auditable?
Dahlmann: Yeah that's, actually, that's
something I just discussed in the
international debate as well, that there
might, that machines might be more ethical
than humans could be. And well, of course
they won't just start raping women because
they want to but you can program them to
do this. So you just you shift the
problems really. And also maybe these
machines don't get angry but they don't
show compassion either so if you are there
and your potential target they just won't
stop they will just kill you and do not
think once think about this. So you have
to really look at both sides there I guess.
Herald: Thanks. So we switch over
to microphone 3, please.
Mic 3: Thanks for the talk. Regarding
autonomous cars, self-driving cars,
there's a similar discussion going on
regarding the ethics. How should a car
react in a case of an accident? Should it
protect people outside people, inside,
what are the laws? So there is another
discussion there. Do you work with people
in this area or is this is there any
collaboration?
Dahlmann: Maybe there's less collaboration
than one might think there is. I think
there is. Of course, we we monitor this
debate as well and yeah we think about the
possible applications of the outcomes for
example from this German ethical
commission on self-driving cars for our
work. But I'm a bit torn there because
when you talk about weapons, they are
designed to kill people and cars mostly
are not. So with this ethical committee
you want to avoid killing people or decide
what happens when this accident occurs. So
they are a bit different but of course
yeah you can learn a lot from both
discussions and we aware of that.
Herald: Thanks. Then we're gonna go over
in the back, microphone number 2, please.
Mic 2: Also from me thanks again for this
talk and infusing all this professionalism
into the debate because some of the
surroundings of our, so to say ours
scenery, they like to protest against very
specific things like for example the
Rammstein air base and in my view that's a
bit misguided if you just go out and
protest in a populistic way without
involving these points of expertise that
you offer. And so, thanks again for that.
And then my question: How would you
propose that protests progress and develop
themselves to a higher level to be on the
one hand more effective and on the other
hand more considerate of what is at stake
on all the levels and on
all sides involved?
Dahlmann: Yeah well, first, the Rammstein
issue is completely, actually a completely
different topic. It's drone warfare,
remotely piloted drones, so there are a
lot of a lot of problems with this and
we're starting killings but it's not about
lethal autonomous weapons in particular.
Well if you want to be a part of this
international debate, there's of course
this campaign to stop killer robots and
they have a lot of really good people and
a lot of resources, sources, literature
and things like that to really educate
yourself what's going on there, so that
would be a starting point. And then yeah
just keep talking to scientists about
this and find out where we see the
problems and I mean it's always helpful
for scientists to to talk to people in the
field, so to say. So yeah, keep talking.
Herald: Thanks for that. And the
signal angel signaled that we have
something from the internet.
Signal Angel: Thank you. Question from
IRC: Aren't we already in a killer robot
world? The bot net can attack a nuclear
power plant for example. What do you think?
Dahlmann: I really didn't understand a
word, I'm sorry.
Herald: I didn't understand that as well,
so can you speak closer to
the microphone, please?
Signal Angel: Yes. Aren't we already in a
killer robot world?
Herald: Sorry, that doesn't work. Sorry.
Sorry, we stop that here, we can't hear it
over here. Sorry.
Signal Angel: Okay.
Herald: We're gonna switch over to
microphone two now, please.
Mic 2: I have one little question. So in
your talk, you were focusing on the
ethical questions related to lethal
weapons. Are you aware of ongoing
discussions regarding the ethical aspects
of the design and implementation of less
than lethal autonomous weapons for crowd
control and similar purposes?
Dahlmann: Yeah actually within the CCW,
every term of this Lethal Autonomous
Weapon Systems is disputed also the
"lethal" aspect and for the regulation
that might be easier to focus on this for
now because less than lethal weapons come
with their own problems and the question
if they are ethical and if they can, if
IHL applies to them but I'm not really
deep into this discussion. So I'll just
have to leave it there.
Herald: Thanks and back here to microphone
one, please.
Mic 1: Hi. Thank you for the talk very
much. My question is in the context of the
decreasing cost of both, the hardware and
software, over the next say 20, 40 years.
Outside of a nation-state context like
private forces or non nation-state actors
gaining use of these weapons, do things
like the UN convention or the campaign to
stop killer robots apply are they
considering private individuals trying to
leverage these against others?
Dahlmann: Not sure what the campaign says
about this, I'm not a member there. The
the CCW mostly focuses on international
humanitarian law which is important but I
think it's it's not broad enough. So
questions like proliferation and all this
is connected to your question and not
really or probably won't be part of
regulation there. It's discussed only on
the edges of the of the debates and
negotiations there but it doesn't seem to
be a really issue there.
Mic 1: Thanks.
Herald: And over to microphone six,
please.
Mic 6: Thank you. I have a question as a
researcher: Do you know how far the
development has gone already? So how
transparent or intransparent is your look
into what is being developed and
researched on the side of militaria
working, military people working with
autonomous weapons and developing them?
Dahlmann: Well, for me it's quite
intransparent because I only have only
access to public publicly available
sources so I don't really know what what's
going on behind closed doors in the
military or in the industry there. Of
course you can you can monitor the
civilian applications or developments
which can tell a lot about the the state
of the art and for example the DARPA
the American Development Agency, they
published sometimes a call for papers,
that's not the term, but there you can see
where in which areas they are interested
in then for example they really like this
idea of autonomous killer bug that can
act in swarms and monitor or even kill
people and things like that. So yeah we
try to piece it, piece it together in
our work.
Herald: We do have a little bit more time,
are you okay to answer more questions?
Dahlmann: Sure.
Herald: Then we're gonna switch over to
microphone three, please.
Mic 3: Yes, hello. I think we are living
already in a world of Leathal Autonomous
Weapon Systems if you think about these
millions of landmines which are operating.
And so the question is: Shouldn't it be
possible to ban these weapon systems the
same way as land mines that are already
banned by several countries so just
include them in that definition? And
because the arguments should be very
similar.
Dahlmann: Yeah it does, it does come to
mind of course because these mines are
just lying around there and no one's
interacting when you step on them and
boom! But they are, well it depends, it
depends first of all a bit of your
definition of autonomy. So some say
autonomous is when you act in dynamic
situations and the other ones would be
automated and things like that and I think
this autonomy aspect, I really don't want
to find, don't want to find define
autonomy here really but this this action
in more dynamic spaces and the aspect of
machine learning and all these things,
they are way more complex and they bring
different problems than just land mines.
Landmines are problematic, anti-personnel
mines are banned for good reasons but they
don't have the same problems I think. So
it won't be, I don't think it won't be
sufficient to just put the LAWS in there,
the Lethal Autonomous Weapons.
Herald: Thank you very much. I can't see
anyone else queuing up so therefore, Anja,
thank you very much it's your applause!
applause
and once again my apologies that
that didn't work
34c3 outro
subtitles created by c3subtitles.de
in the year 2018. Join, and help us!