Go Deeper: Transfer Learning - TensorFlow and Deep Learning Singapore

Speaker: Martin Andrews

Event Page: https://www.meetup.com/TensorFlow-and-Deep-Learning-Singapore/events/237032130/

Produced by Engineers.SG

English subtitles by: Sindhu Shetty

What we hope to do with this meetup

is have something, given the spread of
the questionnaire results

we hope to do something which is kind of

for people who don't know what
deep learning is

and want an introduction to TensorFlow

but also something which is more of a

like a crowd pleaser or something
which is more cutting edge

I am not going to say that this
thing is particularly cutting edge

because once we saw the responses,
we dialed things down a bit

But there will be more cutting edge stuff

and maybe we start to do other meetups
events in other formats

So it could be like we have
an experts' paper meeting

or we could split it now we can see
the size of people, size of the crowd

Anyway, let me talk a little bit about
going deeper with transfer learning

Unfortunately, this is something
some of you people

would have seen me do before

This is the first time I have
done it in tensorflow

and let me just explain that

Before, I have been programming this stuff

in Theano with the
Lasagna layers thing on top

and Theano is a research-based
deep learning framework, out of Montreal

but what I have concluded
since last summer

is that TensorFlow 's probably the winner
of this framework race

at least for foreseeable future

with all this nice industrial stuff

I should be retooling into TensorFlow

That's what I am taking the opportunity
to do for this

So, about me, sorry here we go

I have come up through finance,
startups and stuff

I took a year out basically in 2014
just for fun

I have been doing serious kind of
natural language processing since then

Basically, the overview for this
something more challenging talk

which will probably be 20 mins, 30 mins
depending on how it goes

I want to take a state-of-the-art
TensorFlow model

I want to solve a problem that
it wasn't trained for

And I am going to be using
deep learning as a component

of my solution rather than the
primary focus of what I am trying to build

So this is, in a way more of an industrial
or commercial kind of application

for what's going on here

So the goal for this kind of problem is

I want to distinguish pictures
of classic and modern sports cars

you will see some pictures of
classic and modern cars a bit later

It's not that easy to say what
the difference is

obviously, it could be
different types of images

and it could be lots of
different classes

I am just doing a very simple
two class thing

but it's complicated images

what I want to do is

I want to have a very small training time

so I don't want to be retraining
some huge network

Particularly, I have only got
in this case, 20 training examples

So I am not gonna do any fantastic
million image training

I have got 20 images with me

and I also want to be able to
put this in production

so I can just run it as a component of
something else

Basically, one of the things that is
carrying the deep learning world forward

is an image classification task
called ImageNet

this has been a competition where

they have 15 million labeled images
from 22,000 categories

and you can see some of them here

if we go for this.
this is a picture of a hotdog in a bun

and here are some of the categories

which will be some food I don't know

these are hotdogs, lots of
different pictures of hotdogs

lots of different pictures of cheeseburgers

lots of different pictures of plates

so the task for ImageNet is to classify

for any given, any one of these images

which of a thousand different
categories it is from

and it used to be that people could
score adequately well

and were making incremental changes in

how well they can do this

but the deep learning people came along

and kind of tore this to shreds

and Google came up with GoogLeNet

what we are actually going to use here,
back in 2014

suddenly, this stuff is now being done
with further iterations

of this kind of thing,
better than humans can

So the way you can measure whether
someone is better than humans

is, you take a human and see
whether it beats him

the question there is
are there labeling errors

there you need a committee of humans

so the way they label these things is

by running it on Mechanical Turk and

asking people what category is this
cheeseburger in

The network we are going to use here

is the 2014 state-of-the-art GoogLeNet,
also called Inception version 1

The nice thing about this is that

there is an existing model
already trained for this task

and it's available for download
it's all free

and there are lots of different
models out there

there's a model zoo for TensorFlow

So, what I have on my machine

and this is a small model,
it's a 20 megabytes kind of model

So it is not a very big model

Inception 4 is a 200 MB kind of model
which is a bit heavy

I am working here on my laptop

you are gonna see it working in real-time

and the trick here is instead of
a softmax layer at the end

I will show you the diagram, it should be
clear to anyone who's following along

instead of using logits to get me
the probablilities

I am going to strip that away

and I am going to train
a support vector machine

to distinguish these classes

I am not going to retrain the
Inception network at all

I am going to just use it as a component

strip off the top classification piece

and replace it with an SVM

Now, SVMs are pretty well understood

here I am just using Inception
as a featurizer for images

So here's a network picture

Basically, this is what the ImageNet
network is designed for

you put in an image at the bottom

there is this black box which is the
Inception network

which is a bunch of CNNs or
convolutional neural networks

followed by Dense network

followed by these logits

and these logits layers is essentially
the same as the 0 to 10

that Sam had for his digits, 1 to 1000
for the different classes for ImageNet

To actually get the ImageNet output

it uses a softmax function and
then chooses the highest one of these

to give you this is the class
that this is in

What I am going to do is
I am going to ignore this

neat piece of classification technology
that they have got

let's say we use these outputs as inputs
to SVM and just treat these as features

Now if we pick out one of these

this class could be cheeseburger
and this class could be parrot

and this other class could be Husky dog

there is all sorts of classes in here

but basically what I will be doing is that

I will be extracting out the features
of these photos

saying how much of this photo
is like a parrot

how much of this is like a Husky dog

Now it turns out that modern cars and
classic cars can be distinguised that way

Let me go to some code

Ok this code is all up on GitHub

Can everyone see this enough

So basically, I am pulling in TensorFlow

I pull in this model

Here is what the Inception architecture is

It feeds forward this way,
here you put your image

it goes through lots and lots of
convolutional layers

all the way up to the end
with softmax and the output

So having done that, what I will do is

actually I have a download
for the checkpoint

this is the checkpoint here which
is a tar file, I have it locally stored

It doesn't download it now

but it is all there, even the
big models are there up from Google

so they have retrained these

so the Inception thing takes about a week

to retrain on a bunch of,
it could be 64 GPUs

so you don't really want to be
training this thing on your own

you also need the ImageNet training set

it is a 140 GB file
which is no fun to download

what I am doing here is basically
there is also an Inception library

which is part of the TF-Slim
this thing is desinged such that

it already knows the network
it can preload it

this has loaded it,
I can get some labels

This is loading up the ImageNet labels

I need to know which location
corresponds to which class like the digits

Here we are going through
basically the same steps

as the MNIST example in that
we reset the default graph

we create a placeholder which is
where my images are going to go

this is as an input
but from this image input

I am then going to do some TensorFlow steps

because TensorFlow
has various preprocessing

or graphics handling commands

because a lot of this stuff
works with images

so there's all sorts of clipping
and rotating stuff

so it can preprocess these images

I am also going to pull out a numpy image

so I can see what it is actually looking at

here with this Inception version 1

I am going to pull in the entire
Inception version 1 model

My net function rather than being
just picks and random weights

is gonna be assigned this
from this checkpoint

when I run the init thing from my graph

or in my session, it won't initialize
everything from random

it will initialize everything from disk

so this will define the model

and now let's proceed

one of the issues with having this
on a nice TensorFlow graph

is it just says input, Inception1, output

so there's a big block there
you can delve into it if you want

let me just show you
let's go back a bit

So this is the code
behind the Inception1 model

so this is actually smaller than the
Inception2 and Inception3

basically, we have a kind of a base
Inception piece, just this

and these are combined together

and this is a detailed model put together
by many smart people in 2014

it's got much more complicated since then

fortunately, they have written the code
and we don't have to

So here what I am gonna do is
I am gonna load an example image

just to show you
one of the the things here is

TensorFlow in order to become efficient
wants to do the loading itself

So in order to get this pumping
information through

it wants you to set up hues of images

it will then handle the whole ingestion
process itself

the problem with that is
it's kind of complicated to do

in a Jupyter notebook right here

so here I am going to do
the very simplest thing

which is load a numpy image
and stuff the numpy image in

but what TensorFlow would love me to do

is create , as you see in this one

create a file name queue and it will

then run the queue, do the matching

and do all of this stuff itself

because then it can lay it out across
potentially distributed cluster

and do everything just right

here I do kind of the simple read the image

so this image is a tensor
which is 224 by 224 by RGB

this is kind of sanity check
what kind of numbers I got in the corner

and then what I am gonna do is

i am going to crop out the
middle section of it

this happens to be the right size already

basically if you got odd shapes

you need to think about
how am I gonna do it

am I going to pad it
what do you do

because in order to make this efficient

TensorFlow wants to lay it out without
all this variability in image size

one set of parameters and it's then going
to blast it across your GPU

so let's just run this thing

so now we have defined the network

here I am going to pick a session
here I am going to init the session

it loads the data, and then I am going

to pick up the numpy image and the
probabilities from the top layer

I am just gonna show it

here is the image
this is image I pulled out of the disk

you can see here the probabilities,
the highest probability is Tabby cat

which is good, it's also interesting that

the second in line things are Tiger cat,
Egyptian cat, lynx

so it's got a fair idea that it is a cat
in particular, it is getting it right

ok so this is the same diagram
we have had before

what you have seen is this going in this
black box, coming out and telling us

the probabilities here, so what we are
now gonna do is

from the image to the black box and
just learn a bunch of features

let me just show you this on disk

so I have a cars directory here

and inside this thing,

I have surprisingly little data

In this directory, I just have a
bunch of car images

and I have two sets of images

one of which is called classic
and the other is called modern

so I basically I picked some
photos off Flickr

I put these into two separate directories

I am going to use those directory names

as the classification for these images

In the upper directory here
I got a bunch of test images

which I don't know the labels for

this picks out the list of classic , there
is a classic and a modern directory

I am gonna go through every file
in this directory

I am gonna crop it, I am gonna find
the logits level which is

all the classes and then I am just gonna
add these to features

So basically I am gonna do something
like a scikit-learn model

I am gonna fit SVM

so basically, this is featurizing
all these pictures

so here we go with the training data

here's some training

classic cars,
it went through the classic directory

modern cars,
it went through the modern directory

it's thinking hard

what I am gonna do now is
build SVM over those features

jump to 21:36

I restarted this thing

the actual training for this SVM thing
takes that long,

this is very quick, essentially 20 images
worth of a thousand features

so there was no big training loop to do

then I can run this on the actual models
in the directory, in the test set

so here this is images that it has never
seen before

it thinks that this is a modern car

this one it thinks is a classic car,
this one is classified as modern

so this is actually doing quite a good job
out of just 10 examples of each

it actually thinks this one is modern
it's not a sports car but anyway

so this is showing that the SVM we trained

can classify based on the features that
Inception is producing because

Inception understands "understands"
what images are about

so if I go back to here,
code is on GitHub

conclusions okay, this thing really works

we didn't have to train
a deep neural network

we could plug this TensorFlow model
into an existing pipeline

and this is actually something where

the TensorFlow Summit has something
to say about these pipelines

because not only are they talking
about deep learning

they are talking about the whole
cloud-based learning

and setting up proper processes

I guess, time for questions quickly

we can then do the
TensorFlow Summit wrap-up

"I am assuming that there is no
backpropagation here"

This includes no backpropagation

"End result is a feature"

I am just assuming that Inception,
you can imagine if the ImageNet thing

had focused more on products,
it could be even better

if it focused on man-made things

The ImageNet training set has an awful
lot of dogs in it, not that many cats

So, on the other hand it may be that
it has quite a lot of flowers

or maybe that it is saying I like this car
as modern car

because it's got petals for wheels

whereas the other one, the classic cars
tend to have round things for wheels

So it is abstractly doing this

It doesn't know about sports cars or
what they look like

But it does know about curves

"So for SVM, you don't use
TensorFlow anymore ?"

No, basically I have used TensorFlow to
create some features

Now, I don't want to throw it away

because hopefully I have got
a streaming process where

more and more images are chugged
through this thing

could not hear the question properly

There is an example code called
TensorFlow for poets

where they actually say that,
let's load up one of these networks

and then we will do some fine tuning

there you get involved in tuning
these neurons with some gradient descent

and you are taking some steps
and all this kind of thing

maybe you are having broad implications
across the whole network

which could be good if you have got
tons of data and tons of time

but this is a very simple way of just
tricking it to get it done

could not hear the comment properly

it will be a very small network
because SVM is essentially fairly shallow

could not hear the question

TensorFlow even though it has imported
this large Inception network

as far as I am concerned,
I am using a f(x) = y and that's it

but you can inquire what would it say
at this particular level

and these bunches of levels with various
component points along the way

I could take out other levels

I haven't tried it to have a look

There you get more like pictures
worth of features rather than

this string of a 1000 numbers

but each intermediate levels
will be pictures with CNN kind of features

on the other hand, if you want
to play around with this thing

there's this nice stuff called
the DeepDream kind of things

where they try and match images to
being interesting images

then you do the featurizing that looks at
different levels

the highest level is a cat but I want all
local features to be as fishy as possible

then you get like a fish-faced cat

that's the kind of thing you can do with
these kinds of features in models