Return to Video

34C3 - Deep Learning Blindspots

  • 0:00 - 0:10
    34c3 preroll music
  • 0:16 - 0:18
    Herald: ...and I will let Katherine take
    the stage now.
  • 0:19 - 0:21
    Katharine Jarmul, kjam: Awesome! Well,
    thank you so much for the introduction and
  • 0:21 - 0:25
    thank you so much for being here, taking
    your time. I know that Congress is really
  • 0:25 - 0:30
    exciting, so I really appreciate you
    spending some time with me today. It's my
  • 0:30 - 0:34
    first ever Congress, so I'm also really
    excited and I want to meet new people. So
  • 0:34 - 0:40
    if you wanna come say hi to me later, I'm
    somewhat friendly, so we can maybe be
  • 0:40 - 0:45
    friends later. Today what we're going to
    talk about is deep learning blind spots or
  • 0:45 - 0:50
    how to fool "artificial intelligence". I
    like to put "artificial intelligence" in
  • 0:50 - 0:55
    quotes, because.. yeah, we'll talk about
    that, but I think it should be in quotes.
  • 0:55 - 1:00
    And today we're going to talk a little bit
    about deep learning, how it works and how
  • 1:00 - 1:08
    you can maybe fool it. So I ask us: Is AI
    becoming more intelligent?
  • 1:08 - 1:11
    And I ask this because when I open a
    browser and, of course, often it's Chrome
  • 1:11 - 1:17
    and Google is already prompting me
    for what I should look at
  • 1:17 - 1:20
    and it knows that I work with machine
    learning, right?
  • 1:20 - 1:24
    And these are the headlines
    that I see every day:
  • 1:24 - 1:29
    "Are Computers Already Smarter Than
    Humans?"
  • 1:29 - 1:32
    If so, I think we could just pack up and
    go home, right?
  • 1:32 - 1:36
    Like, we fixed computers,
    right? If a computer is smarter than me,
  • 1:36 - 1:40
    then I already fixed it, we can go home,
    there's no need to talk about computers
  • 1:40 - 1:48
    anymore, let's just move on with life. But
    that's not true, right? We know, because
  • 1:48 - 1:51
    we work with computers and we know how
    stupid computers are sometimes. They're
  • 1:51 - 1:56
    pretty bad. Computers do only what we tell
    them to do, generally, so I don't think a
  • 1:56 - 2:01
    computer can think and be smarter than me.
    So with the same types of headlines that
  • 2:01 - 2:12
    you see this, then you also see this: And
    yeah, so Apple recently released their
  • 2:12 - 2:18
    face ID and this unlocks your phone with
    your face and it seems like a great idea,
  • 2:18 - 2:22
    right? You have a unique face, you have a
    face, nobody else can take your face. But
  • 2:22 - 2:28
    unfortunately what we find out about
    computers is that they're awful sometimes,
  • 2:28 - 2:32
    and for these women.. for this Chinese
    woman that owned an iPhone,
  • 2:32 - 2:36
    her coworker was able to unlock her phone.
  • 2:36 - 2:39
    And I think Hendrick and Karin
    talked about, if you were here for the
  • 2:39 - 2:42
    last talk ("Beeinflussung durch künstliche
    Intelligenz"). We have a lot of problems
  • 2:42 - 2:46
    in machine learning and one of them is
    stereotypes and prejudice that are within
  • 2:46 - 2:52
    our training data or within our minds that
    leak into our models. And perhaps they
  • 2:52 - 2:58
    didn't do adequate training data on
    determining different features of Chinese
  • 2:58 - 3:03
    folks. And perhaps it's other problems
    with their model or their training data or
  • 3:03 - 3:08
    whatever they're trying to do. But they
    clearly have some issues, right? So when
  • 3:08 - 3:12
    somebody asked me: "Is AI gonna take over
    the world and is there a super robot
  • 3:12 - 3:17
    that's gonna come and be my new, you know,
    leader or so to speak?" I tell them we
  • 3:17 - 3:22
    can't even figure out the stuff that we
    already have in production. So if we can't
  • 3:22 - 3:26
    even figure out the stuff we already have
    in production, I'm a little bit less
  • 3:26 - 3:33
    worried of the super robot coming to kill
    me. That said, unfortunately the powers
  • 3:33 - 3:38
    that be, the powers that be a lot of times
    they believe in this and they believe
  • 3:38 - 3:45
    strongly in "artificial intelligence" and
    machine learning. They're collecting data
  • 3:45 - 3:51
    every day about you and me and everyone
    else. And they're gonna use this data to
  • 3:51 - 3:56
    build even better models. This is because
    the revolution that we're seeing now in
  • 3:56 - 4:02
    machine learning has really not much to do
    with new algorithms or architectures. It
  • 4:02 - 4:10
    has a lot more to do with heavy compute
    and with massive, massive data sets. And
  • 4:10 - 4:16
    the more that we have training data of
    petabytes per 24 hours or even less, the
  • 4:16 - 4:23
    more we're able to essentially fix up the
    parts that don't work so well. The
  • 4:23 - 4:26
    companies that we see here are companies
    that are investing heavily in machine
  • 4:26 - 4:31
    learning and AI. Part of how they're
    investing heavily is, they're collecting
  • 4:31 - 4:38
    more and more data about you and me and
    everyone else. Google and Facebook, more
  • 4:38 - 4:43
    than 1 billion active users. I was
    surprised to know that in Germany the
  • 4:43 - 4:48
    desktop search traffic for Google is
    higher than most of the rest of the world.
  • 4:48 - 4:53
    And for Baidu they're growing with the
    speed that broadband is available. And so,
  • 4:53 - 4:57
    what we see is, these people are
    collecting this data and they also are
  • 4:57 - 5:03
    using new technologies like GPUs and TPUs
    in new ways to parallelize workflows
  • 5:03 - 5:09
    and with this they're able to mess up
    less, right? They're still messing up, but
  • 5:09 - 5:15
    they mess up slightly less. And they're
    not going to get uninterested in this
  • 5:15 - 5:21
    topic, so we need to kind of start to
    prepare how we respond to this type of
  • 5:21 - 5:26
    behavior. One of the things that has been
    a big area of research, actually also for
  • 5:26 - 5:30
    a lot of these companies, is what we'll
    talk about today and that's adversarial
  • 5:30 - 5:37
    machine learning. But the first thing that
    we'll start with is what is behind what we
  • 5:37 - 5:44
    call AI. So most of the time when you
    think of AI or something like Siri and so
  • 5:44 - 5:49
    forth, you are actually potentially
    talking about an old-school rule-based
  • 5:49 - 5:54
    system. This is a rule, like you say a
    particular thing and then Siri is like:
  • 5:54 - 5:58
    "Yes, I know how to respond to this". And
    we even hard program these types of things
  • 5:58 - 6:03
    in, right? That is one version of AI, is
    essentially: It's been pre-programmed to
  • 6:03 - 6:09
    do and understand certain things. Another
    form that usually, for example for the
  • 6:09 - 6:13
    people that are trying to build AI robots
    and the people that are trying to build
  • 6:13 - 6:17
    what we call "general AI", so this is
    something that can maybe learn like a
  • 6:17 - 6:20
    human, they'll use reinforcement learning.
  • 6:20 - 6:22
    I don't specialize in reinforcement
    learning.
  • 6:22 - 6:26
    But what it does is it essentially
    tries to reward you for
  • 6:26 - 6:32
    behaviour that you're expected to do. So
    if you complete a task, you get a a
  • 6:32 - 6:36
    cookie. You complete two other tasks, you
    get two or three more cookies depending on
  • 6:36 - 6:42
    how important the task is. And this will
    help you learn how to behave to get more
  • 6:42 - 6:46
    points and it's used a lot in robots and
    gaming and so forth. And I'm not really
  • 6:46 - 6:49
    going to talk about that today because
    most of that is still not really something
  • 6:49 - 6:55
    that you or I interact with. Well, what I
    am gonna talk about today is neural
  • 6:55 - 7:00
    networks, or as some people like to call
    them "deep learning", right? So deep
  • 7:00 - 7:04
    learning 1: The neural network versus deep
    learning battle awhile ago. So here's an
  • 7:04 - 7:10
    example neural network: we have an input
    layer and that's where we essentially make
  • 7:10 - 7:15
    a quantitative version of whatever our
    data is. So we need to make it into
  • 7:15 - 7:20
    numbers. Then we have a hidden layer and
    we might have multiple hidden layers. And
  • 7:20 - 7:24
    depending on how deep our network is, or a
    network inside a network, right, which is
  • 7:24 - 7:28
    possible. We might have very much
    different layers there and they may even
  • 7:28 - 7:34
    act in cyclical ways. And then that's
    where all the weights and the variables
  • 7:34 - 7:39
    and the learning happens. So that has..
    holds a lot of information and data that
  • 7:39 - 7:44
    we eventually want to train there. And
    finally we have an output layer. And
  • 7:44 - 7:48
    depending on the network and what we're
    trying to do the output layer can vary
  • 7:48 - 7:52
    between something that looks like the
    input, like for example if we want to
  • 7:52 - 7:56
    machine translate, then I want the output
    to look like the input, right, I want it
  • 7:56 - 8:00
    to just be in a different language, or the
    output could be a different class. It can
  • 8:00 - 8:06
    be, you know, this is a car or this is a
    train and so forth. So it really depends
  • 8:06 - 8:11
    what you're trying to solve, but the
    output layer gives us the answer. And how
  • 8:11 - 8:17
    we train this is, we use backpropagation.
    Backpropagation is nothing new and neither
  • 8:17 - 8:21
    is one of the most popular methods to do
    so, which is called stochastic gradient
  • 8:21 - 8:26
    descent. What we do when we go through
    that part of the training, is we go from
  • 8:26 - 8:30
    the output layer and we go backwards
    through the network. That's why it's
  • 8:30 - 8:35
    called backpropagation, right? And as we
    go backwards through the network, in the
  • 8:35 - 8:39
    most simple way, we upvote and downvote
    what's working and what's not working. So
  • 8:39 - 8:43
    we say: "oh you got it right, you get a
    little bit more importance", or "you got
  • 8:43 - 8:46
    it wrong, you get a little bit less
    importance". And eventually we hope
  • 8:46 - 8:50
    over time, that they essentially correct
    each other's errors enough that we get a
  • 8:50 - 8:58
    right answer. So that's a very general
    overview of how it works and the cool
  • 8:58 - 9:03
    thing is: Because it works that way, we
    can fool it. And people have been
  • 9:03 - 9:08
    researching ways to fool it for quite some
    time. So I give you a brief overview of
  • 9:08 - 9:13
    the history of this field, so we can kind
    of know where we're working from and maybe
  • 9:13 - 9:19
    hopefully then where we're going to. In
    2005 was one of the first most important
  • 9:19 - 9:25
    papers to approach adversarial learning
    and it was written by a series of
  • 9:25 - 9:30
    researchers and they wanted to see, if
    they could act as an informed attacker and
  • 9:30 - 9:34
    attack a linear classifier. So this is
    just a spam filter and they're like can I
  • 9:34 - 9:38
    send spam to my friend? I don't know why
    they would want to do this, but: "Can I
  • 9:38 - 9:43
    send spam to my friend, if I tried testing
    out a few ideas?" And what they were able
  • 9:43 - 9:48
    to show is: Yes, rather than just, you
    know, trial and error which anybody can do
  • 9:48 - 9:52
    or a brute force attack of just like send
    a thousand emails and see what happens,
  • 9:52 - 9:56
    they were able to craft a few algorithms
    that they could use to try and find
  • 9:56 - 10:03
    important words to change, to make it go
    through the spam filter. In 2007 NIPS,
  • 10:03 - 10:08
    which is a very popular machine learning
    conference, had one of their first all-day
  • 10:08 - 10:13
    workshops on computer security. And when
    they did so, they had a bunch of different
  • 10:13 - 10:17
    people that were working on machine
    learning in computer security: From
  • 10:17 - 10:21
    malware detection, to network intrusion
    detection, to of course spam. And they
  • 10:21 - 10:25
    also had a few talks on this type of
    adversarial learning. So how do you act as
  • 10:25 - 10:30
    an adversary to your own model? And then
    how do you learn how to counter that
  • 10:30 - 10:36
    adversary? In 2013 there was a really
    great paper that got a lot of people's
  • 10:36 - 10:40
    attention called "Poisoning Attacks
    against Support Vector Machines". Now
  • 10:40 - 10:45
    support vector machines are essentially
    usually a linear classifier and we use
  • 10:45 - 10:50
    them a lot to say, "this is a member of
    this class, that, or another", when we
  • 10:50 - 10:55
    pertain to text. So I have a text and I
    want to know what the text is about or I
  • 10:55 - 10:59
    want to know if it's a positive or
    negative sentiment, a lot of times I'll
  • 10:59 - 11:05
    use a support vector machine. We call them
    SVM's as well. Battista Biggio was the
  • 11:05 - 11:08
    main researcher and he has actually
    written quite a lot about these poisoning
  • 11:08 - 11:16
    attacks and he poisoned the training data.
    So for a lot of these systems, sometimes
  • 11:16 - 11:21
    they have active learning. This means, you
    or I, when we classify our emails as spam,
  • 11:21 - 11:26
    we're helping train the network. So he
    poisoned the training data and was able to
  • 11:26 - 11:32
    show that by poisoning it in a particular
    way, that he was able to then send spam
  • 11:32 - 11:38
    email because he knew what words were then
    benign, essentially. He went on to study a
  • 11:38 - 11:43
    few other things about biometric data if
    you're interested in biometrics. But then
  • 11:43 - 11:49
    in 2014 Christian Szegedy, Ian Goodfellow,
    and a few other main researchers at Google
  • 11:49 - 11:55
    Brain released "Intriguing Properties of
    Neural Networks." That really became the
  • 11:55 - 12:00
    explosion of what we're seeing today in
    adversarial learning. And what they were
  • 12:00 - 12:05
    able to do, is they were able to say "We
    believe there's linear properties of these
  • 12:05 - 12:09
    neural networks, even if they're not
    necessarily linear networks.
  • 12:09 - 12:16
    And we believe we can exploit them to fool
    them". And they first introduced then the
  • 12:16 - 12:23
    fast gradient sign method, which we'll
    talk about later today. So how does it
  • 12:23 - 12:29
    work? First I want us to get a little bit
    of an intuition around how this works.
  • 12:29 - 12:35
    Here's a graphic of gradient descent. And
    in gradient descent we have this vertical
  • 12:35 - 12:40
    axis is our cost function. And what we're
    trying to do is: We're trying to minimize
  • 12:40 - 12:47
    cost, we want to minimize the error. And
    so when we start out, we just chose random
  • 12:47 - 12:52
    weights and variables, so all of our
    hidden layers, they just have maybe random
  • 12:52 - 12:57
    weights or random distribution. And then
    we want to get to a place where the
  • 12:57 - 13:02
    weights have meaning, right? We want our
    network to know something, even if it's
  • 13:02 - 13:09
    just a mathematical pattern, right? So we
    start in the high area of the graph, or
  • 13:09 - 13:14
    the reddish area, and that's where we
    started, we have high error there. And
  • 13:14 - 13:21
    then we try to get to the lowest area of
    the graph, or here the dark blue that is
  • 13:21 - 13:27
    right about here. But sometimes what
    happens: As we learn, as we go through
  • 13:27 - 13:33
    epochs and training, we're moving slowly
    down and hopefully we're optimizing. But
  • 13:33 - 13:37
    what we might end up in instead of this
    global minimum, we might end up in the
  • 13:37 - 13:44
    local minimum which is the other trail.
    And that's fine, because it's still zero
  • 13:44 - 13:50
    error, right? So we're still probably
    going to be able to succeed, but we might
  • 13:50 - 13:56
    not get the best answer all the time. What
    adversarial tries to do in the most basic
  • 13:56 - 14:02
    of ways, it essentially tries to push the
    error rate back up the hill for as many
  • 14:02 - 14:08
    units as it can. So it essentially tries
    to increase the error slowly through
  • 14:08 - 14:15
    perturbations. And by disrupting, let's
    say, the weakest links like the one that
  • 14:15 - 14:19
    did not find the global minimum but
    instead found a local minimum, we can
  • 14:19 - 14:23
    hopefully fool the network, because we're
    finding those weak spots and we're
  • 14:23 - 14:26
    capitalizing on them, essentially.
  • 14:31 - 14:34
    So what does an adversarial example
    actually look like?
  • 14:34 - 14:37
    You may have already seen this
    because it's very popular on the
  • 14:37 - 14:45
    Twittersphere and a few other places, but
    this was a series of researches at MIT. It
  • 14:45 - 14:51
    was debated whether you could do adverse..
    adversarial learning in the real world. A
  • 14:51 - 14:57
    lot of the research has just been a still
    image. And what they were able to show:
  • 14:57 - 15:03
    They created a 3D-printed turtle. I mean
    it looks like a turtle to you as well,
  • 15:03 - 15:10
    correct? And this 3D-printed turtle by the
    Inception Network, which is a very popular
  • 15:10 - 15:17
    computer vision network, is a rifle and it
    is a rifle in every angle that you can
  • 15:17 - 15:22
    see. And the way they were able to do this
    and, I don't know the next time it goes
  • 15:22 - 15:26
    around you can see perhaps, and it's a
    little bit easier on the video which I'll
  • 15:26 - 15:30
    have posted, I'll share at the end, you
    can see perhaps that there's a slight
  • 15:30 - 15:36
    discoloration of the shell. They messed
    with the texture. By messing with this
  • 15:36 - 15:40
    texture and the colors they were able to
    fool the neural network, they were able to
  • 15:40 - 15:45
    activate different neurons that were not
    supposed to be activated. Units, I should
  • 15:45 - 15:51
    say. So what we see here is, yeah, it can
    be done in the real world, and when I saw
  • 15:51 - 15:56
    this I started getting really excited.
    Because, video surveillance is a real
  • 15:56 - 16:03
    thing, right? So if we can start fooling
    3D objects, we can perhaps start fooling
  • 16:03 - 16:08
    other things in the real world that we
    would like to fool.
  • 16:08 - 16:12
    applause
  • 16:12 - 16:19
    kjam: So why do adversarial examples
    exist? We're going to talk a little bit
  • 16:19 - 16:24
    about some things that are approximations
    of what's actually happening, so please
  • 16:24 - 16:28
    forgive me for not being always exact, but
    I would rather us all have a general
  • 16:28 - 16:34
    understanding of what's happening. Across
    the top row we have an input layer and
  • 16:34 - 16:39
    these images to the left, we can see, are
    the source images and this source image is
  • 16:39 - 16:43
    like a piece of farming equipment or
    something. And on the right we have our
  • 16:43 - 16:49
    guide image. This is what we're trying to
    get the network to see we want it to
  • 16:49 - 16:55
    missclassify this farm equipment as a pink
    bird. So what these researchers did is
  • 16:55 - 16:59
    they targeted different layers of the
    network. And they said: "Okay, we're going
  • 16:59 - 17:02
    to use this method to target this
    particular layer and we'll see what
  • 17:02 - 17:08
    happens". And so as they targeted these
    different layers you can see what's
  • 17:08 - 17:12
    happening on the internal visualization.
    Now neural networks can't see, right?
  • 17:12 - 17:18
    They're looking at matrices of numbers but
    what we can do is we can use those
  • 17:18 - 17:27
    internal values to try and see with our
    human eyes what they are learning. And we
  • 17:27 - 17:31
    can see here clearly inside the network,
    we no longer see the farming equipment,
  • 17:31 - 17:40
    right? We see a pink bird. And this is not
    visible to our human eyes. Now if you
  • 17:40 - 17:44
    really study and if you enlarge the image
    you can start to see okay there's a little
  • 17:44 - 17:48
    bit of pink here or greens, I don't know
    what's happening, but we can still see it
  • 17:48 - 17:57
    in the neural network we have tricked. Now
    people don't exactly know yet why these
  • 17:57 - 18:03
    blind spots exist. So it's still an area
    of active research exactly why we can fool
  • 18:03 - 18:09
    neural networks so easily. There are some
    prominent researchers that believe that
  • 18:09 - 18:14
    neural networks are essentially very
    linear and that we can use this simple
  • 18:14 - 18:21
    linearity to misclassify to jump into
    another area. But there are others that
  • 18:21 - 18:25
    believe that there's these pockets or
    blind spots and that we can then find
  • 18:25 - 18:28
    these blind spots where these neurons
    really are the weakest links and they
  • 18:28 - 18:33
    maybe even haven't learned anything and if
    we change their activation then we can
  • 18:33 - 18:38
    fool the network easily. So this is still
    an area of active research and let's say
  • 18:38 - 18:44
    you're looking for your thesis, this would
    be a pretty neat thing to work on. So
  • 18:44 - 18:49
    we'll get into just a brief overview of
    some of the math behind the most popular
  • 18:49 - 18:56
    methods. First we have the fast gradient
    sign method and that is was used in the
  • 18:56 - 19:00
    initial paper and now there's been many
    iterations on it. And what we do is we
  • 19:00 - 19:05
    have our same cost function, so this is
    the same way that we're trying to train
  • 19:05 - 19:13
    our network and it's trying to learn. And
    we take the gradient sign of that and if
  • 19:13 - 19:16
    you can think, it's okay, if you're not
    used to doing vector calculus, and
  • 19:16 - 19:20
    especially not without a pen and paper in
    front of you, but what you think we're
  • 19:20 - 19:24
    doing is we're essentially trying to
    calculate some approximation of a
  • 19:24 - 19:30
    derivative of the function. And this can
    kind of tell us, where is it going. And if
  • 19:30 - 19:37
    we know where it's going, we can maybe
    anticipate that and change. And then to
  • 19:37 - 19:41
    create the adversarial images, we then
    take the original input plus a small
  • 19:41 - 19:49
    number epsilon times that gradient's sign.
    For the Jacobian Saliency Map, this is a
  • 19:49 - 19:55
    newer method and it's a little bit more
    effective, but it takes a little bit more
  • 19:55 - 20:02
    compute. This Jacobian Saliency Map uses a
    Jacobian matrix and if you remember also,
  • 20:02 - 20:08
    and it's okay if you don't, a Jacobian
    matrix will look at the full derivative of
  • 20:08 - 20:12
    a function, so you take the full
    derivative of a cost function
  • 20:12 - 20:18
    at that vector, and it gives you a matrix
    that is a pointwise approximation,
  • 20:18 - 20:23
    if the function is differentiable
    at that input vector. Don't
  • 20:23 - 20:28
    worry you can review this later too. But
    the Jacobian matrix then we use to create
  • 20:28 - 20:33
    this saliency map the same way where we're
    essentially trying some sort of linear
  • 20:33 - 20:39
    approximation, or pointwise approximation,
    and we then want to find two pixels that
  • 20:39 - 20:44
    we can perturb that cause the most
    disruption. And then we continue to the
  • 20:44 - 20:49
    next. Unfortunately this is currently a
    O(n²) problem, but there's a few people
  • 20:49 - 20:54
    that are trying to essentially find ways
    that we can approximate this and make it
  • 20:54 - 21:01
    faster. So maybe now you want to fool a
    network too and I hope you do, because
  • 21:01 - 21:07
    that's what we're going to talk about.
    First you need to pick a problem or a
  • 21:07 - 21:13
    network type you may already know. But you
    may want to investigate what perhaps is
  • 21:13 - 21:19
    this company using, what perhaps is this
    method using and do a little bit of
  • 21:19 - 21:24
    research, because that's going to help
    you. Then you want to research state-of-
  • 21:24 - 21:29
    the-art methods and this is like a typical
    research statement that you have a new
  • 21:29 - 21:32
    state-of-the-art method, but the good news
    is is that the state-of-the-art two to
  • 21:32 - 21:38
    three years ago is most likely in
    production or in systems today. So once
  • 21:38 - 21:44
    they find ways to speed it up, some
    approximation of that is deployed. And a
  • 21:44 - 21:48
    lot of times these are then publicly
    available models, so a lot of times, if
  • 21:48 - 21:51
    you're already working with the deep
    learning framework they'll come
  • 21:51 - 21:56
    prepackaged with a few of the different
    popular models, so you can even use that.
  • 21:56 - 22:01
    If you're already building neural networks
    of course you can build your own. An
  • 22:01 - 22:06
    optional step, but one that might be
    recommended, is to fine-tune your model
  • 22:06 - 22:11
    and what this means is to essentially take
    a new training data set, maybe data that
  • 22:11 - 22:15
    you think this company is using or that
    you think this network is using, and
  • 22:15 - 22:19
    you're going to remove the last few layers
    of the neural network and you're going to
  • 22:19 - 22:25
    retrain it. So you essentially are nicely
    piggybacking on the work of the pre
  • 22:25 - 22:31
    trained model and you're using the final
    layers to create finesse. This essentially
  • 22:31 - 22:37
    makes your model better at the task that
    you have for it. Finally then you use a
  • 22:37 - 22:40
    library, and we'll go through a few of
    them, but some of the ones that I have
  • 22:40 - 22:46
    used myself is cleverhans, DeepFool and
    deep-pwning, and these all come with nice
  • 22:46 - 22:52
    built-in features for you to use for let's
    say the fast gradient sign method, the
  • 22:52 - 22:57
    Jacobian saliency map and a few other
    methods that are available. Finally it's
  • 22:57 - 23:02
    not going to always work so depending on
    your source and your target, you won't
  • 23:02 - 23:06
    always necessarily find a match. What
    researchers have shown is it's a lot
  • 23:06 - 23:11
    easier to fool a network that a cat is a
    dog than it is to fool in networks that a
  • 23:11 - 23:16
    cat is an airplane. And this is just like
    we can make these intuitive, so you might
  • 23:16 - 23:22
    want to pick an input that's not super
    dissimilar from where you want to go, but
  • 23:22 - 23:28
    is dissimilar enough. And you want to test
    it locally and then finally test the one
  • 23:28 - 23:38
    for the highest misclassification rates on
    the target network. And you might say
  • 23:38 - 23:44
    Katharine, or you can call me kjam, that's
    okay. You might say: "I don't know what
  • 23:44 - 23:50
    the person is using", "I don't know what
    the company is using" and I will say "it's
  • 23:50 - 23:57
    okay", because what's been proven: You can
    attack a blackbox model, you do not have
  • 23:57 - 24:02
    to know what they're using, you do not
    have to know exactly how it works, you
  • 24:02 - 24:07
    don't even have to know their training
    data, because what you can do is if it
  • 24:07 - 24:13
    has.. okay, addendum it has to have some
    API you can interface with. But if it has
  • 24:13 - 24:18
    an API you can interface with or even any
    API you can interact with, that uses the
  • 24:18 - 24:25
    same type of learning, you can collect
    training data by querying the API. And
  • 24:25 - 24:29
    then you're training your local model on
    that data that you're collecting. So
  • 24:29 - 24:33
    you're collecting the data, you're
    training your local model, and as your
  • 24:33 - 24:37
    local model gets more accurate and more
    similar to the deployed black box that you
  • 24:37 - 24:43
    don't know how it works, you are then
    still able to fool it. And what this paper
  • 24:43 - 24:50
    proved, Nicolas Papanov and a few other
    great researchers, is that with usually
  • 24:50 - 24:57
    less than six thousand queries they were
    able to fool the network between 84% and 97% certainty
  • 24:59 - 25:03
    And what the same group
    of researchers also studied is the ability
  • 25:03 - 25:09
    to transfer the ability to fool one
    network into another network and they
  • 25:09 - 25:15
    called that transfer ability. So I can
    take a certain type of network and I can
  • 25:15 - 25:19
    use adversarial examples against this
    network to fool a different type of
  • 25:19 - 25:26
    machine learning technique. Here we have
    their matrix, their heat map, that shows
  • 25:26 - 25:33
    us exactly what they were able to fool. So
    we have across the left-hand side here the
  • 25:33 - 25:38
    source machine learning technique, we have
    deep learning, logistic regression, SVM's
  • 25:38 - 25:43
    like we talked about, decision trees and
    K-nearest-neighbors. And across the bottom
  • 25:43 - 25:47
    we have the target machine learning, so
    what were they targeting. They created the
  • 25:47 - 25:51
    adversaries with the left hand side and
    they targeted across the bottom. We
  • 25:51 - 25:57
    finally have an ensemble model at the end.
    And what they were able to show is like,
  • 25:57 - 26:03
    for example, SVM's and decision trees are
    quite easy to fool, but logistic
  • 26:03 - 26:08
    regression a little bit less so, but still
    strong, for deep learning and K-nearest-
  • 26:08 - 26:13
    neighbors, if you train a deep learning
    model or a K-nearest-neighbor model, then
  • 26:13 - 26:18
    that performs fairly well against itself.
    And so what they're able to show is that
  • 26:18 - 26:23
    you don't necessarily need to know the
    target machine and you don't even have to
  • 26:23 - 26:28
    get it right, even if you do know, you can
    use a different type of machine learning
  • 26:28 - 26:30
    technique to target the network.
  • 26:34 - 26:39
    So we'll
    look at six lines of Python here and in
  • 26:39 - 26:45
    these six lines of Python I'm using the
    cleverhans library and in six lines of
  • 26:45 - 26:52
    Python I can both generate my adversarial
    input and I can even predict on it. So if
  • 26:52 - 27:02
    you don't code Python, it's pretty easy to
    learn and pick up. And for example here we
  • 27:02 - 27:07
    have Keras and Keras is a very popular
    deep learning library in Python, it
  • 27:07 - 27:12
    usually works with a theano or a
    tensorflow backend and we can just wrap
  • 27:12 - 27:19
    our model, pass it to the fast gradient
    method, class and then set up some
  • 27:19 - 27:25
    parameters, so here's our epsilon and a
    few extra parameters, this is to tune our
  • 27:25 - 27:31
    adversary, and finally we can generate our
    adversarial examples and then predict on
  • 27:31 - 27:40
    them. So in a very small amount of Python
    we're able to target and trick a network.
  • 27:41 - 27:46
    If you're already using tensorflow or
    Keras, it already works with those libraries.
  • 27:49 - 27:53
    Deep-pwning is one of the first
    libraries that I heard about in this space
  • 27:53 - 27:58
    and it was presented at Def Con in 2016
    and what it comes with is a bunch of
  • 27:58 - 28:03
    tensorflow built-in code. It even comes
    with a way that you can train the model
  • 28:03 - 28:07
    yourself, so it has a few different
    models, a few different convolutional
  • 28:07 - 28:12
    neural networks and these are
    predominantly used in computer vision.
  • 28:12 - 28:18
    It also however has a semantic model and I
    normally work in NLP and I was pretty
  • 28:18 - 28:24
    excited to try it out. What it comes built
    with is the Rotten Tomatoes sentiment, so
  • 28:24 - 28:30
    this is Rotten Tomatoes movie reviews that
    try to learn is it positive or negative.
  • 28:30 - 28:35
    So the original text that I input in, when
    I was generating my adversarial networks
  • 28:35 - 28:42
    was "more trifle than triumph", which is a
    real review and the adversarial text that
  • 28:42 - 28:46
    it gave me was "jonah refreshing haunting
    leaky"
  • 28:49 - 28:53
    ...Yeah.. so I was able to fool my network
  • 28:53 - 28:58
    but I lost any type of meaning and
    this is really the problem when we think
  • 28:58 - 29:04
    about how we apply adversarial learning to
    different tasks is, it's easy for an image
  • 29:04 - 29:09
    if we make a few changes for it to retain
    its image, right? It's many, many pixels,
  • 29:09 - 29:14
    but when we start going into language, if
    we change one word and then another word
  • 29:14 - 29:19
    and another word or maybe we changed all
    of the words, we no longer understand as
  • 29:19 - 29:23
    humans. And I would say this is garbage
    in, garbage out, this is not actual
  • 29:23 - 29:29
    adversarial learning. So we have a long
    way to go when it comes to language tasks
  • 29:29 - 29:33
    and being able to do adversarial learning
    and there is some research in this, but
  • 29:33 - 29:37
    it's not really advanced yet. So hopefully
    this is something that we can continue to
  • 29:37 - 29:42
    work on and advance further and if so we
    need to support a few different types of
  • 29:42 - 29:47
    networks that are more common in NLP than
    they are in computer vision.
  • 29:50 - 29:55
    There's some other notable open-source libraries that
    are available to you and I'll cover just a
  • 29:55 - 30:00
    few here. There's a "Vanderbilt
    computational economics research lab" that
  • 30:00 - 30:04
    has adlib and this allows you to do
    poisoning attacks. So if you want to
  • 30:04 - 30:09
    target training data and poison it, then
    you can do so with that and use scikit-
  • 30:09 - 30:17
    learn. DeepFool allows you to do the fast
    gradient sign method, but it tries to do
  • 30:17 - 30:22
    smaller perturbations, it tries to be less
    detectable to us humans.
  • 30:23 - 30:28
    It's based on Theano, which is another library that I believe uses Lua as well as Python.
  • 30:30 - 30:34
    "FoolBox" is kind of neat because I only
    heard about it last week, but it collects
  • 30:34 - 30:39
    a bunch of different techniques all in one
    library and you could use it with one
  • 30:39 - 30:43
    interface. So if you want to experiment
    with a few different ones at once, I would
  • 30:43 - 30:47
    recommend taking a look at that and
    finally for something that we'll talk
  • 30:47 - 30:54
    about briefly in a short period of time we
    have "Evolving AI Lab", which release a
  • 30:54 - 31:00
    fooling library and this fooling library
    is able to generate images that you or I
  • 31:00 - 31:05
    can't tell what it is, but that the neural
    network is convinced it is something.
  • 31:05 - 31:10
    So this we'll talk about maybe some
    applications of this in a moment, but they
  • 31:10 - 31:14
    also open sourced all of their code and
    they're researchers, who open sourced
  • 31:14 - 31:20
    their code, which is always very exciting.
    As you may have known from some of the
  • 31:20 - 31:26
    research I already cited, most of the
    studies and the research in this area has
  • 31:26 - 31:30
    been on malicious attacks. So there's very
    few people trying to figure out how to do
  • 31:30 - 31:34
    this for what I would call benevolent
    purposes. Most of them are trying to act
  • 31:34 - 31:40
    as an adversary in the traditional
    computer security sense. They're perhaps
  • 31:40 - 31:44
    studying spam filters and how spammers can
    get by them. They're perhaps looking at
  • 31:44 - 31:49
    network intrusion or botnet-attacks and so
    forth. They're perhaps looking at self-
  • 31:49 - 31:53
    driving cars so and I know that was
    referenced earlier as well at Henrick and
  • 31:53 - 31:58
    Karen's talk, they're perhaps trying to
    make a yield sign look like a stop sign or
  • 31:58 - 32:03
    a stop sign look like a yield sign or a
    speed limit, and so forth, and scarily
  • 32:03 - 32:08
    they are quite successful at this. Or
    perhaps they're looking at data poisoning,
  • 32:08 - 32:12
    so how do we poison the model so we render
    it useless? In a particular context, so we
  • 32:12 - 32:18
    can utilize that. And finally for malware.
    So what a few researchers were able to
  • 32:18 - 32:23
    show is, by just changing a few things in
    the malware they were able to upload their
  • 32:23 - 32:26
    malware to Google Mail and send it to
    someone and this was still fully
  • 32:26 - 32:32
    functional malware. In that same sense
    there's the malGAN project, which uses a
  • 32:32 - 32:39
    generative adversarial network to create
    malware that works, I guess. So there's a
  • 32:39 - 32:43
    lot of research of these kind of malicious
    attacks within adversarial learning.
  • 32:45 - 32:52
    But what I wonder is how might we use this for
    good. And I put "good" in quotation marks,
  • 32:52 - 32:56
    because we all have different ethical and
    moral systems we use. And what you may
  • 32:56 - 33:00
    decide is ethical for you might be
    different. But I think as a community,
  • 33:00 - 33:05
    especially at a conference like this,
    hopefully we can converge on some ethical
  • 33:05 - 33:10
    privacy concerned version of using these
    networks.
  • 33:13 - 33:21
    So I've composed a few ideas and I hope that this is just a starting list of a longer conversation.
  • 33:23 - 33:30
    One idea is that we can perhaps use this type of adversarial learning to fool surveillance.
  • 33:31 - 33:36
    As surveillance affects you and I it even
    disproportionately affects people that
  • 33:36 - 33:42
    most likely can't be here. So whether or
    not we're personally affected, we can care
  • 33:42 - 33:46
    about the many lives that are affected by
    this type of surveillance. And we can try
  • 33:46 - 33:50
    and build ways to fool surveillance
    systems.
  • 33:51 - 33:52
    Stenography:
  • 33:52 - 33:55
    So we could potentially, in a world where more and more people
  • 33:55 - 33:59
    have less of a private way of sending messages to one another
  • 33:59 - 34:03
    We can perhaps use adversarial learning to send private messages.
  • 34:04 - 34:08
    Adware fooling: So
    again, where I might have quite a lot of
  • 34:08 - 34:14
    privilege and I don't actually see ads
    that are predatory on me as much, there is
  • 34:14 - 34:19
    a lot of people in the world that face
    predatory advertising. And so how can we
  • 34:19 - 34:24
    help those problems by developing
    adversarial techniques?
  • 34:25 - 34:27
    Poisoning your own private data:
  • 34:27 - 34:31
    This depends on whether you
    actually need to use the service and
  • 34:31 - 34:35
    whether you like how the service is
    helping you with the machine learning, but
  • 34:35 - 34:40
    if you don't care or if you need to
    essentially have a burn box of your data.
  • 34:40 - 34:46
    Then potentially you could poison your own
    private data. Finally, I want us to use it
  • 34:46 - 34:51
    to investigate deployed models. So even
    if we don't actually need a use for
  • 34:51 - 34:56
    fooling this particular network, the more
    we know about what's deployed and how we
  • 34:56 - 35:00
    can fool it, the more we're able to keep
    up with this technology as it continues to
  • 35:00 - 35:05
    evolve. So the more that we're practicing,
    the more that we're ready for whatever
  • 35:05 - 35:10
    might happen next. And finally I really
    want to hear your ideas as well. So I'll
  • 35:10 - 35:14
    be here throughout the whole Congress and
    of course you can share during the Q&A
  • 35:14 - 35:17
    time. If you have great ideas, I really
    want to hear them.
  • 35:21 - 35:26
    So I decided to play around a little bit with some of my ideas.
  • 35:27 - 35:33

    And I was convinced perhaps that I could make Facebook think I was a cat.
  • 35:33 - 35:36
    This is my goal. Can Facebook think I'm a cat?
  • 35:38 - 35:41
    Because nobody really likes Facebook. I
    mean let's be honest, right?
  • 35:42 - 35:44
    But I have to be on it because my mom messages me there
  • 35:44 - 35:46
    and she doesn't use the email anymore.
  • 35:46 - 35:48
    So I'm on Facebook. Anyways.
  • 35:48 - 35:55
    So I used a pre-trained Inception model and Keras and I fine-tuned the layers.
  • 35:55 - 35:57
    And I'm not a
    computer vision person really. But it
  • 35:57 - 36:02
    took me like a day of figuring out how
    computer vision people transfer their data
  • 36:02 - 36:06
    into something I can put inside of a
    network figure that out and I was able to
  • 36:06 - 36:12
    quickly train a model and the model could
    only distinguish between people and cats.
  • 36:12 - 36:15
    That's all the model knew how to do. I
    give it a picture it says it's a person or
  • 36:15 - 36:20
    it's a cat. I actually didn't try just
    giving it an image of something else, it
  • 36:20 - 36:25
    would probably guess it's a person or a
    cat maybe, 50/50, who knows. What I did
  • 36:25 - 36:32
    was, I used an image of myself and
    eventually I had my fast gradient sign
  • 36:32 - 36:38
    method, I used cleverhans, and I was able
    to slowly increase the epsilon and so the
  • 36:38 - 36:44
    epsilon as it's low, you and I can't see
    the perturbations, but also the network
  • 36:44 - 36:49
    can't see the perturbations. So we need to
    increase it, and of course as we increase
  • 36:49 - 36:53
    it, when we're using a technique like
    FGSM, we are also increasing the noise
  • 36:53 - 37:01
    that we see. And when I got 2.21 epsilon
    and I kept uploading it to Facebook and
  • 37:01 - 37:02
    Facebook kept saying: "Yeah, do you want
    to tag yourself?" and I'm like:
  • 37:02 - 37:04
    "no Idon't, I'm just testing".
  • 37:05 - 37:11
    Finally I got deployed to an epsilon and Facebook no longer knew I was a face
  • 37:11 - 37:15
    So I was just a
    book, I was a cat book, maybe.
  • 37:15 - 37:20
    applause
  • 37:21 - 37:25
    kjam: So, unfortunately, as we see, I
    didn't actually become a cat, because that
  • 37:25 - 37:31
    would be pretty neat. But I was able to
    fool it. I spoke with the computer visions
  • 37:31 - 37:35
    specialists that I know and she actually
    works in this and I was like: "What
  • 37:35 - 37:39
    methods do you think Facebook was using?
    Did I really fool the neural network or
  • 37:39 - 37:43
    what did I do?" And she's convinced most
    likely that they're actually using a
  • 37:43 - 37:48
    statistical method called Viola-Jones,
    which takes a look at the statistical
  • 37:48 - 37:53
    distribution of your face and tries to
    guess if there's really a face there. But
  • 37:53 - 37:59
    what I was able to show: transferability.
    That is, I can use my neural network even
  • 37:59 - 38:05
    to fool this statistical model, so now I
    have a very noisy but happy photo on FB
  • 38:09 - 38:14
    Another use case potentially is
    adversarial stenography and I was really
  • 38:14 - 38:19
    excited reading this paper. What this
    paper covered and they actually released
  • 38:19 - 38:23
    the library, as I mentioned. They study
    the ability of a neural network to be
  • 38:23 - 38:26
    convinced that something's there that's
    not actually there.
  • 38:27 - 38:30
    And what they used, they used the MNIST training set.
  • 38:30 - 38:33
    I'm sorry, if that's like a trigger word
  • 38:33 - 38:38
    if you've used MNIST a million times, then
    I'm sorry for this, but what they use is
  • 38:38 - 38:43
    MNIST, which is zero through nine of
    digits, and what they were able to show
  • 38:43 - 38:49
    using evolutionary networks is they were
    able to generate things that to us look
  • 38:49 - 38:53
    maybe like art and they actually used it
    on the CIFAR data set too, which has
  • 38:53 - 38:57
    colors, and it was quite beautiful. Some
    of what they created in fact they showed
  • 38:57 - 39:04
    in a gallery. And what the network sees
    here is the digits across the top. They
  • 39:04 - 39:12
    see that digit, they are more than 99%
    convinced that that digit is there and
  • 39:12 - 39:15
    what we see is pretty patterns or just
    noise.
  • 39:17 - 39:20
    When I was reading this paper I was thinking,
  • 39:20 - 39:24
    how can we use this to send
    messages to each other that nobody else
  • 39:24 - 39:29
    will know is there? I'm just sending
    really nice.., I'm an artist and this is
  • 39:29 - 39:35
    my art and I'm sharing it with my friend.
    And in a world where I'm afraid to go home
  • 39:35 - 39:42
    because there's a crazy person in charge
    and I'm afraid that they might look at my
  • 39:42 - 39:47
    phone, in my computer, and a million other
    things and I just want to make sure that
  • 39:47 - 39:52
    my friend has my pin number or this or
    that or whatever. I see a use case for my
  • 39:52 - 39:56
    life, but again I leave a fairly
    privileged life, there are other people
  • 39:56 - 40:02
    where their actual life and livelihood and
    security might depend on using a technique
  • 40:02 - 40:06
    like this. And I think we could use
    adversarial learning to create a new form
  • 40:06 - 40:07
    of stenography.
  • 40:11 - 40:17
    Finally I cannot impress
    enough that the more information we have
  • 40:17 - 40:21
    about the systems that we interact with
    every day, that our machine learning
  • 40:21 - 40:25
    systems, that our AI systems, or whatever
    you want to call it, that our deep
  • 40:25 - 40:30
    networks, the more information we have,
    the better we can fight them, right. We
  • 40:30 - 40:34
    don't need perfect knowledge, but the more
    knowledge that we have, the better an
  • 40:34 - 40:41
    adversary we can be. I thankfully now live
    in Germany and if you are also a European
  • 40:41 - 40:47
    resident: We have GDPR, which is the
    general data protection regulation and it
  • 40:47 - 40:56
    goes into effect in May of 2018. We can
    use gdpr to make requests about our data,
  • 40:56 - 41:00
    we can use GDPR to make requests about
    machine learning systems that we interact
  • 41:00 - 41:08
    with, this is a right that we have. And in
    recital 71 of the GDPR it states: "The
  • 41:08 - 41:13
    data subject should have the right to not
    be subject to a decision, which may
  • 41:13 - 41:18
    include a measure, evaluating personal
    aspects relating to him or her which is
  • 41:18 - 41:23
    based solely on automated processing and
    which produces legal effects concerning
  • 41:23 - 41:28
    him or her or similarly significantly
    affects him or her, such as automatic
  • 41:28 - 41:34
    refusal of an online credit application or
    e-recruiting practices without any human
  • 41:34 - 41:39
    intervention." And I'm not a lawyer and I
    don't know how this will be implemented
  • 41:39 - 41:44
    and it's a recital, so we don't even know,
    if it will be in force the same way, but
  • 41:44 - 41:51
    the good news is: Pieces of this same
    sentiment are in the actual amendments and
  • 41:51 - 41:56
    if they're in the amendments, then we can
    legally use them. And what it also says
  • 41:56 - 42:00
    is, we can ask companies to port our data
    other places, we can ask companies to
  • 42:00 - 42:04
    delete our data, we can ask for
    information about how our data is
  • 42:04 - 42:09
    processed, we can ask for information
    about what different automated decisions
  • 42:09 - 42:16
    are being made, and the more we all here
    ask for that data, the more we can also
  • 42:16 - 42:21
    share that same information with people
    worldwide. Because the systems that we
  • 42:21 - 42:25
    interact with, they're not special to us,
    they're the same types of systems that are
  • 42:25 - 42:31
    being deployed everywhere in the world. So
    we can help our fellow humans outside of
  • 42:31 - 42:36
    Europe by being good caretakers and using
    our rights to make more information
  • 42:36 - 42:42
    available to the entire world and to use
    this information, to find ways to use
  • 42:42 - 42:46
    adversarial learning to fool these types
    of systems.
  • 42:48 - 42:56
    applause
  • 42:57 - 43:03
    So how else might we be able to harness
    this for good? I cannot focus enough on
  • 43:03 - 43:08
    GDPR and our right to collect more
    information about the information they're
  • 43:08 - 43:14
    already collecting about us and everyone
    else. So use it, let's find ways to share
  • 43:14 - 43:18
    the information we gain from it. So I
    don't want it to just be that one person
  • 43:18 - 43:21
    requests it and they learn something. Se
    have to find ways to share this
  • 43:21 - 43:28
    information with one another. Test low-
    tech ways. I'm so excited about the maker
  • 43:28 - 43:33
    space here and maker culture and other
    low-tech or human-crafted ways to fool
  • 43:33 - 43:38
    networks. We can use adversarial learning
    perhaps to get good ideas on how to fool
  • 43:38 - 43:43
    networks, to get lower tech ways. What if
    I painted red pixels all over my face?
  • 43:43 - 43:49
    Would I still be recognized? Would I not?
    Let's experiment with things that we learn
  • 43:49 - 43:54
    from adversarial learning and try to find
    other lower-tech solutions to the same problem
  • 43:55 - 44:00
    Finally. or nearly finally, we
    need to increase the research beyond just
  • 44:00 - 44:04
    computer vision. Quite a lot of
    adversarial learning has been only in
  • 44:04 - 44:08
    computer vision and while I think that's
    important and it's also been very
  • 44:08 - 44:12
    practical, because we can start to see how
    we can fool something, we need to figure
  • 44:12 - 44:16
    out natural language processing, we need
    to figure out other ways that machine
  • 44:16 - 44:20
    learning systems are being used, and we
    need to come up with clever ways to fool them.
  • 44:22 - 44:26
    Finally, spread the word! So I don't
    want the conversation to end here, I don't
  • 44:26 - 44:31
    want the conversation to end at Congress,
    I want you to go back to your hacker
  • 44:31 - 44:37
    collective, your local CCC, the people
    that you talk with, your co-workers and I
  • 44:37 - 44:41
    want you to spread the word. I want you to
    do workshops on adversarial learning, I
  • 44:41 - 44:48
    want more people to not treat this AI as
    something mystical and powerful, because
  • 44:48 - 44:52
    unfortunately it is powerful, but it's not
    mystical! So we need to demystify this
  • 44:52 - 44:57
    space, we need to experiment, we need to
    hack on it and we need to find ways to
  • 44:57 - 45:02
    play with it and spread the word to other
    people. Finally, I really want to hear
  • 45:02 - 45:10
    your other ideas and before I leave today
    have to say a little bit about why I
  • 45:10 - 45:16
    decided to join the resiliency track this
    year. I read about the resiliency track
  • 45:16 - 45:22
    and I was really excited. It spoke to me.
    And I said I want to live in a world
  • 45:22 - 45:27
    where, even if there's an entire burning
    trash fire around me, I know that there
  • 45:27 - 45:32
    are other people that I care about, that I
    can count on, that I can work with to try
  • 45:32 - 45:38
    and at least protect portions of our
    world. To try and protect ourselves, to
  • 45:38 - 45:44
    try and protect people that do not have as
    much privilege. So, what I want to be a
  • 45:44 - 45:49
    part of, is something that can use maybe
    the skills I have and the skills you have
  • 45:49 - 45:57
    to do something with that. And your data
    is a big source of value for everyone.
  • 45:57 - 46:03
    Any free service you use, they are selling
    your data. OK, I don't know that for a
  • 46:03 - 46:08
    fact, but it is very certain, I feel very
    certain about the fact that they're most
  • 46:08 - 46:13
    likely selling your data. And if they're
    selling your data, they might also be
  • 46:13 - 46:18
    buying your data. And there is a whole
    market, that's legal, that's freely
  • 46:18 - 46:23
    available, to buy and sell your data. And
    they make money off of that, and they mine
  • 46:23 - 46:29
    more information, and make more money off
    of that, and so forth. So, I will read a
  • 46:29 - 46:35
    little bit of my opinions that I put forth
    on this. Determine who you share your data
  • 46:35 - 46:42
    with and for what reasons. GDPR and data
    portability give us European residents
  • 46:42 - 46:44
    stronger rights than most of the world.
  • 46:45 - 46:48
    Let's use them. Let's choose privacy
  • 46:48 - 46:53
    concerned ethical data companies over
    corporations that are entirely built on
  • 46:53 - 46:58
    selling ads. Let's build start-ups,
    organizations, open-source tools and
  • 46:58 - 47:06
    systems that we can be truly proud of. And
    let's port our data to those.
  • 47:06 - 47:15
    Applause
  • 47:15 - 47:19
    Herald: Amazing. We have,
    we have time for a few questions.
  • 47:19 - 47:22
    K.J.: I'm not done yet, sorry, it's fine.
    Herald: I'm so sorry.
  • 47:22 - 47:25
    K.J.: Laughs It's cool.
    No big deal.
  • 47:25 - 47:32
    So, machine learning. Closing remarks is
    brief round up. Closing remarks. There is
  • 47:32 - 47:35
    that machine learning is not very
    intelligent. I think artificial
  • 47:35 - 47:39
    intelligence is a misnomer in a lot of
    ways, but this doesn't mean that people
  • 47:39 - 47:44
    are going to stop using it. In fact
    there's very smart, powerful, and rich
  • 47:44 - 47:50
    people that are investing more than ever
    in it. So it's not going anywhere. And
  • 47:50 - 47:54
    it's going to be something that
    potentially becomes more dangerous over
  • 47:54 - 47:59
    time. Because as we hand over more of
    these to these systems, it could
  • 47:59 - 48:04
    potentially control more and more of our
    lives. We can use, however, adversarial
  • 48:04 - 48:09
    machine learning techniques to find ways
    to fool "black box" networks. So we can
  • 48:09 - 48:14
    use these and we know we don't have to
    have perfect knowledge. However,
  • 48:14 - 48:19
    information is powerful. And the more
    information that we do have, the more were
  • 48:19 - 48:26
    able to become a good GDPR based
    adversary. So please use GDPR and let's
  • 48:26 - 48:31
    discuss ways where we can share
    information. Finally, please support open-
  • 48:31 - 48:36
    source tools and research in this space,
    because we need to keep up with where the
  • 48:36 - 48:42
    state of the art is. So we need to keep
    ourselves moving and open in that way. And
  • 48:42 - 48:47
    please, support ethical data companies. Or
    start one. If you come to me and you say
  • 48:47 - 48:50
    "Katharine, I'm going to charge you this
    much money, but I will never sell your
  • 48:50 - 48:57
    data. And I will never buy your data." I
    would much rather you handle my data. So I
  • 48:57 - 49:03
    want us, especially those within the EU,
    to start a new economy around trust, and
  • 49:03 - 49:13
    privacy, and ethical data use.
    Applause
  • 49:13 - 49:16
    Thank you very much.
    Thank you.
  • 49:16 - 49:18
    Herald: OK. We still have time for a few
    questions.
  • 49:18 - 49:20
    K.J.: No, no, no. No worries, no worries.
    Herald: Less than the last time I walked
  • 49:20 - 49:24
    up here, but we do.
    K.J.: Yeah, now I'm really done.
  • 49:24 - 49:28
    Herald: Come up to one of the mics in the
    front section and raise your hand. Can we
  • 49:28 - 49:32
    take a question from mic one.
    Question: Thank you very much for the very
  • 49:32 - 49:38
    interesting talk. One impression that I
    got during the talk was, with the
  • 49:38 - 49:42
    adversarial learning approach aren't we
    just doing pen testing and Quality
  • 49:42 - 49:48
    Assurance for the AI companies they're
    just going to build better machines.
  • 49:48 - 49:53
    Answer: That's a very good question and of
    course most of this research right now is
  • 49:53 - 49:57
    coming from those companies, because
    they're worried about this. What, however,
  • 49:57 - 50:02
    they've shown is, they don't really have a
    good way to fool, to learn how to fool
  • 50:02 - 50:09
    this. Most likely they will need to use a
    different type of network, eventually. So
  • 50:09 - 50:13
    probably, whether it's the blind spots or
    the linearity of these networks, they are
  • 50:13 - 50:18
    easy to fool and they will have to come up
    with a different method for generating
  • 50:18 - 50:25
    something that is robust enough to not be
    tricked. So, to some degree yes, its a
  • 50:25 - 50:29
    cat-and-mouse game, right. But that's why
    I want the research and the open source to
  • 50:29 - 50:33
    continue as well. And I would be highly
    suspect if they all of a sudden figure out
  • 50:33 - 50:38
    a way to make a neural network which has
    proven linear relationships, that we can
  • 50:38 - 50:43
    exploit, nonlinear. And if so, it's
    usually a different type of network that's
  • 50:43 - 50:47
    a lot more expensive to train and that
    doesn't actually generalize well. So we're
  • 50:47 - 50:51
    going to really hit them in a way where
    they're going to have to be more specific,
  • 50:51 - 51:00
    try harder, and I would rather do that
    than just kind of give up.
  • 51:00 - 51:03
    Herald: Next one.
    Mic 2
  • 51:03 - 51:08
    Q: Hello. Thank you for the nice talk. I
    wanted to ask, have you ever tried looking
  • 51:08 - 51:15
    at from the other direction? Like, just
    trying to feed the companies falsely
  • 51:15 - 51:22
    classified data. And just do it with so
    massive amounts of data, so that they
  • 51:22 - 51:25
    learn from it at a certain point.
    A: Yes, that's these poisoning attacks. So
  • 51:25 - 51:30
    when we talk about poison attacks, we are
    essentially feeding bad training data and
  • 51:30 - 51:35
    we're trying to get them to learn bad
    things. Or I wouldn't say bad things, but
  • 51:35 - 51:38
    we're trying to get them to learn false
    information.
  • 51:38 - 51:43
    And that already happens on accident all
    the time so I think the more to we can, if
  • 51:43 - 51:46
    we share information and they have a
    publicly available API, where they're
  • 51:46 - 51:50
    actually actively learning from our
    information, then yes I would say
  • 51:50 - 51:55
    poisoning is a great attack way. And we
    can also share information of maybe how
  • 51:55 - 51:58
    that works.
    So especially I would be intrigued if we
  • 51:58 - 52:02
    can do poisoning for adware and malicious
    ad targeting.
  • 52:02 - 52:07
    Mic 2: OK, thank you.
    Herald: One more question from the
  • 52:07 - 52:12
    internet and then we run out of time.
    K.J. Oh no, sorry
  • 52:12 - 52:14
    Herald: So you can find Katherine after.
    Signal-Angel: Thank you. One question from
  • 52:14 - 52:18
    the internet. What exactly can I do to
    harden my model against adversarial
  • 52:18 - 52:21
    samples?
    K.J.: Sorry?
  • 52:21 - 52:27
    Signal: What exactly can I do to harden my
    model against adversarial samples?
  • 52:27 - 52:33
    K.J.: Not much. What they have shown is,
    that if you train on a mixture of real
  • 52:33 - 52:39
    training data and adversarial data it's a
    little bit harder to fool, but that just
  • 52:39 - 52:45
    means that you have to try more iterations
    of adversarial input. So right now, the
  • 52:45 - 52:52
    recommendation is to train on a mixture of
    adversarial and real training data and to
  • 52:52 - 52:56
    continue to do that over time. And I would
    argue that you need to maybe do data
  • 52:56 - 53:00
    validation on input. And if you do data
    validation on input maybe you can
  • 53:00 - 53:05
    recognize abnormalities. But that's
    because I come from mainly like production
  • 53:05 - 53:09
    levels not theoretical, and I think maybe
    you should just test things, and see if
  • 53:09 - 53:15
    look weird you should maybe not take them
    into the system.
  • 53:15 - 53:19
    Herald: And that's all for the questions.
    I wish we had more time but we just don't.
  • 53:19 - 53:22
    Please give it up for Katharine Jarmul
  • 53:22 - 53:26
    Applause
  • 53:26 - 53:31
    34c3 postroll music
  • 53:31 - 53:48
    subtitles created by c3subtitles.de
    in the year 2019. Join, and help us!
Title:
34C3 - Deep Learning Blindspots
Description:

more » « less
Video Language:
English
Duration:
53:48

English subtitles

Revisions Compare revisions