10.6: Pixel Neighbors - Processing Tutorial

Book: Learning Processing A Beginner's Guide to Programming, Images,Animation, and Interaction

Chapter: 15

Official book website: http://learningprocessing.com/

Twitter: https://twitter.com/shiffman

This video covers how to compare pixels to its neighbors for the purpose of image processing.

This video needs links to source code examples!
This video needs links to other things mentioned!
Please write in the comments what is missing and what would be helpful!

Ok, so here's another important aspect

about image processing that we left out

in the previous video.

(clears throat)

So what did we do?

We had, an image of a frog

we processed every single pixel and we

copied that pixel to our processing display
window and maybe we manipulated it while

copying it, we made it brighter
we changed it's color, we did all sorts of things.

This is, I would say, pixel point processing.

What we are doing is there's a one to one

relationship, for this pixel

change this pixel

for the next pixel, change this pixel

for the next pixel, change the next pixel.

However, most sophisticated image processing

algorithms that you might find in

not to keep using photoshop as an example

photoshop or a ** image processing.

actually process goups of pixels

for example,

let's say this is my frog

pixel

my frog image

and this here is my processing window,

and let's say this is the pixel I want to

set here in my processing window.

What if inorder to set this pixel
I actually

used a whole bunch of neighbouring pixels

around that pixel, so that pixel

is over here

I don't know what am drawing here exactly
umm but I actually took all of the

pixels in five by five matrix around it and
this is the roo.. the color of this pixel

is a function of all of these pixels
combined.

Ahh aah, a classic image processing

algorithm that you might

have used before is a blur.

What if you take a pic.. an image

and blur it making it fuzzier,

less sharp, less in focus.

Well actually if you average all of the pixels

of an area

to set a particular pixel, that's a

blur algorithm.

It is making each pixel

more like it's neighbours

almost ahh bluring it, averaging it,

smoothing it between pixels.

So this is a actually an algorithm

and sharpening it is the opposite

taking each pixel and making it more

different than it's neighbours.

Umm another example
of an algorithm is finding the edges

in an image.
So if I have a silhouette of a person

(chuckles)

this looks nothing like a person

and I don't know..

some weird ghost alien thing

and I wanna find the edges of this person

well let's say this person was wearing

a black shirt,

much like the.. my black shirt here

how would I find the edges?

Well, if this is

if you're looking at an image of me

the pixels that are very different

the white pixels and these black pixels

a pixel that's very different

from it's neighbours is an edge right?

(finger taps his microphone twice)

this pixel over here is very similar to

all it's neighbours.

This pixel right here is quite different

than it's neighbours it must be

on an edge.

So, finding the edges of an image

has to do with looking for pixels that

are very different from their neighbours.

Let's actually see if we can,

implement this really really quickly

in our frog example.

(taps to switch cameras)

So if I come over here, to the frog

and if I look at this froooooog

and what where we doing here?

We were ahh kinda doing this threshold

thing, that's where we last were.

Well, let me say, let's aah

take all this out and

let's take all this out.

Here is a particular pixel location,

what if I want to look at this pixel

relative to the pixel to it's right

or to it's left, or above or below.

Well, what will that pixel be this is

location1 another location might be

(x+1) times y times width

and if you look at this what is

the difference here?

Now am saying, let me look at pixel x + y

xy and let me look at pixel x+1, y

right? Here ahh if am over here and then

am in some image

if this is pixel x - oh you can't see that.

If this is pixel x, y then this right here

right next to it is pixel x+1, y

so ofcourse a more sophisticated algorithm

migh look at x+1 and x-1 and y+1 and y-1

x+2 and y+2 aloot of pixels

in the neigboring area but just to

start really simply let's look at this pixel

and the one next to it.

(taps to switch camera)

So if I'm looking at this pixel and

the one next to it, I can look at both

of the brightness of both of those pixels.

b1 equals frog.pixels[loc1];

b2 equals frog.pixels[loc2];

Aaah those cameras are always shutting off

am gonna fix that in the middle

of this video look I turned black

for a second and am back

and am just going to fix this one

you'll have to bear with me, there's

nothing I can do about this right now.

ohhh come back (taps twice)

Ok, am back I'l edit all that out later

ok so now uhh I could get a

difference which is maybe am just gonna

say the absolute value of (b1-b2)

and let's set the pixel of the window

equal to that difference

noow, oops loc1

so loc1 is kind of my core original

location so I wanna set that individual

pixel but I wanna look at it as a function

the value of that pixel as a function of

two pixels that are next to each other

let's run this and by the way

we got this sad, very difficult, really

traumatic array out of bounds exception

this error is something that will happen

to you as soon as you start doing any

image processing work. What happened?

I tried to look at, investigate,

think about access a picel that doesn't

exist, because at some point I was at a

pixel that was all the way at the edge

of my image and I tried to look at

pixel one more to the edge

which doesn't exist.

So there's alot of strategies around this

problem for right now am just going to

do something which is kind of simple

which is say let me just look at all the

pixels all the way up to one before the
edge

and that will kind of like deal with this

problem but you can think of other things

you could have pixels wrap around

there's lots of other ways you could

approach this problem

but now we can see

you know this isn't the perfect

demonstration but I've kinda really got

something here the the frog by the way

was had a lot of similar colors so am

getting ummm umm

actually am not exactly getting

what I expected I guess the frog actually

didn't have lots of similar colors it's

very different cuz we're getting we're

getting a very bright color everywhere

there. I guess there's a big difference

of all the pixels inside the frog

and the background is very

noisy so there's a couple of things

we're noticing here; one - this isn't

such a great algorithm am only looking

at the pixels to the left coming to the
right and to the left if I looked at the

pixels above and below and diagonally

I might be able to uhh determine

alot more about the image

the other thing is - this particular

image has alot of noise in it

so I'm am there's so much noise that

am not really getting accurate edges

maybe blurring the image first and then

looking at it looking to the edges might
help, but in the end am actually

visually able to see something here

but this wasn't the exact demonstration

I was hoping for but uhh

to.. look at this, maybe try to

oohhh I know what the problem is

huuuuuh (sighs deeply)

am gonna have to re-record this video

but what did I just do here?
I did the thing that I did originally

which is just I looked at the arbitrary
difference between the giant integer value

you're probably all just watching this
like yelling at the screen like you're

doing it wrong, what I wanted to do was

look at the brightness of each one of

this pixels just forgot like I wanna

compare the brightness of each one of

these pixels. This is gonna give us

something much much more umm

oh yeah and it's very hard to see here

but actually this really worked.

I'm really getting the edges I'm really

getting the edges cause the edges of

the frog that's where the pixels are most

different and infact if I want

to be a bit more thoughtful about this

I might just come up with some threshold

like if the pixel values are at all

different then,

draw black otherwise draw white

and now if I run this, we can see

aah I'm really starting to see the

contours of this frog. I think this could

be a better demonstration. So here

we're seeing just a really basic

algorithm for looking at the comparison

between two pixels and finding the edges

now again this was a very simplistic

approach most of the time what you wanna

do wooah haha all that work and this

camera is not even on umm most of

the time what you wanna do is look

at a whole selection of pixels and in fact

this is something that you might not

be aware of this is right there bring up

photoshop for a second if I go to

photoshop and I've got my frog loaded

in here and I go to filter ahh ahh ah

other custom what I dont..

what your actually seeing here

is an and this is hard for you to see

is this are the weights of a pixel

If I wanna make a pixel on the center

much more important than the ones in

the area, that's a sharpening.

I want the pixel to be more different

than it's neighbours I actually ahh

repr.. ahh oops I created very quickly a

blur and you can see here

this this would be a blur, what if

I take all the pixels in a huge neighbour

space, add them all together

divide them by 25 as an average

and you can see here that this

umm frog is now blurred am gonna uncheck

the preview let me kind of zoom in here

umm you can see ahh over here

look over here, that frog is blurred

that frog is not blurred so this

particular algorithm is blurring that

image by adding all the pixels together

and infact umm you know am not

gonna go through the code for this

in detail but we can look at this

right here so if I look at this particular

example this this particular example you can

see here if I zoom in to it that the

pixels inside the rectangle are being

blurred and the pixels outside are not

you can see kinda all the arm above me as

I move the rectangle over am kinda

blurring out that pixel those pixels

if I go back to the code, and I put in

these weights this is actually

sharpening, again I want the pixel

on the center to substract the pixels

around it to be more different than

the pixels around it and if I run this

this is a bit more obvious, we can see

that this is me sharpening that image

now there's alot of noise is happening

here am very zoomed in but you can

see how this is changing the

quality of the pixels so comparing the

pixels and its neighbours this idea of

looking at a pixel and looking at the

ones that are right of it or above or

below is an important powerful

technique in image processing

and we can just you know this video

can go on for hours and hours and

look at every image processing

algorithm that photoshop uses

that anyone uses and we can see this

technique in alot of them. I guess

what am saying here is see what you can

do with this idea.

What if you umm what if you ahh umm try

to recreate this edge detection algorithm

but actually use more than just the two

pixels that are to the right and left

what if you use the pixels above or down

what if you actually were able to

interactly select whether you're looking

at the comparison of vertical pixels

or horizontal pixels there's lots of

possibilities you might explore and try

here and actually something that I

didn't do which I think will be

really interesting to see which I

would like to maybe add to another

video at some point and I.. think of it

as what happens

when we're processing the pixels

this pixel goes here this pixel goes here

what if I start to move the pixel around?

what if I take, what if every time I look

at a pixel I actually offset put it in

some like random location near where

it originally was this will actually give

me almost llike a watery effect because

the pixels start to like look like

they're dispersing or not. This this is an

idea that I'd love to demonstrate

so I think this video was already quite

loong so am not gonna do it here umm

but you can think of this as an excercise

and I'll try to add one in at some point

too that does this.

So here are the things that are missing

umm am gonna upload these so if you're

watching these in the first pass

this is where it's gonna stop

but what I wanna get to in the next set

of videos, number one I wanna look at

moving the pixels around which will

also be relevant to the idea of what

happens if we just sort of like forget

about this way of doing things entirely

it's like this one to one pixel idea

what if I just have a system where am

drawing you know patterns and shapes

to the screen maybe I draw grid of

rectangles or I draw random circles

everywhere or I draw swirls swirly spirals

all over the screen what if just use an

image essentially a database of color

so what if I could like look up colors

doing form a drawing algorithm

create a painting for an image

that's one thing I'm gonna look at

and once we've done that, the next piece

that I wanna look at is what happens

when this images aren't static

what if I'm getting images from a camera

that are changing live real time images or

images from a movie file.

So those are the I guess there's like

three more three/four more videos that

need to come to kinda finish off

all of these concepts.

Ok so send me your feedback, umm I'll

be happy to add more things, delete

things, redo things or just go away not

come back. (laughs) Ok so have good..

enjoy your morning, evening, night.

I don't know when you're watching this
and you're just like eating some cereal

at home that's... something
you might be doing and

enjoy that cereal (laughs) talk to you
later.