Dissecting HDMI (33c3)

33C3 preroll music

Herald-Angel: Tim ‘mithro’ Ansell
has come all the way from Australia

to talk to us about ‘Dissecting HDMI’
and developing an open FPGA-based

capture hardware for sharing
talks outside of the room.

And he will be explaining how
to dissect it. And I’m looking forward

to hearing the talk in a second.
And so please give Tim

a warm round of applause
again. Thank you!

applause

Tim Ansell: Okay, hi, I’m Tim, and
in theory, if my slides change

you would see that.

And I kind of have too many projects.
And I’m gonna be discussing one of them.

This is another project that
I gave a lightning talk on earlier.

If you didn’t see it, it’s an ARM
microcontroller that goes in your USB port.

People wanted to know
when to hack on it.

Tomorrow at 2 PM, apparently.

So, the first that I want to say is
I’m a software developer.

I’m not a hardware designer,
I’m not an FPGA developer,

I’m not a professional in any of that.

I develop software for full time.
So this is my hobby.

As well, this information comes from
a couple of projects that I started

but a lot of other people did the majority
of the work and I’m just telling you

about it because they are too shy
to come up and talk about it themselves.

So a big thank you to all these people
who have helped me in various ways

regarding this. And theses slides –

any of the blue things are links, so
if your playing it along at home

you can get to them by that URL
and click on these things.

And there is probably other people I’ve
forgotten, who are not on this list.

I’m very sorry.

So this title of this talk could be called
“Software guy tries hardware and complains”.

I’ve had a really hard time figuring out
what to call this talk.

And you’ll see some other attempts
of naming this talk better.

So a bit of history.
How do I end up doing HDMI stuff?

So TimVideos is a group of projects
which are trying to make it easy

to record and live-stream user groups
and conferences like this event.

However, we want to do it without
needing the awesome team

that is doing the recording here.

These guys are really, really organized
and professional.

We want to do it where people have
no experience at all with AV,

and just can make it happen.

And so this is how you record
a conference or user group.

I’m gonna be talking about
these two things here,

the HDMI2USB devices that we created.

They are used in our setup both for
camera capture and for capture of slides.

And so HDMI2USB is FOSS hardware
for doing HDMI capture

and actually has a bit of history
with the CCC

because it was inspired by
a speaker who spoke here.

Bunnie spoke on his NeTV board

which was an FPGA Man-in-the-Middle attack

on HDCP-secured links. His talk
is really awesome. It’s gonna be…

that talk is way more technical
than mine and gives you

some really awesome details about the
cool things he did to make that work.

Mine is much more basic. You don’t
need much experience with HDMI

to follow my talk.
Our device works like his does,

except his was deliberately designed
to not allow capture,

our design allows capture.
It effectively man-in-the-middles

the presenters projector,
between the presenters laptop

and the projector, and provides
a high-quality capture out the USB2 port.

I use an FPGA to do that.
This is because

using FPGA makes hardware problems
software problems, and as I said

I’m a software developer, I prefer
software problems to hardware problems.

And the way it kind of works is
it appears as a UVC webcam so that

you can use it with Skype or Hangouts
or any of those things without needing

any drivers on sensible operating
systems like MacOS and Linux.

On Windows you need a driver that
tells it to use the internal driver.

It’s kind of weird.
And also a serial port

because we have the ability to switch
which input goes to which output.

It’s kind of like a matrix.

And so this the open source
hardware we designed.

It’s in KiCAD, you can find it on Github.

I’m quite proud of it.
It’s quite a good little kit.

We don’t use all the features of it yet,
but it’s pretty awesome.

And it’s in use!

We used this technology to capture
at a bunch of conferences:

PyCon in Australia,
linux.conf.au in Australia

– as I said, I’m Australian.

DebConf, though, are not Australian.

They used it in – sorry –
in South Africa, I think.

And there are a whole bunch of
other people around the world

who are using this,
which is pretty awesome.

The main reason I wanted it to be
open source was so that

other people could use them
and learn from them and fix problems,

because there are a lot of problems
we’ve run into.

And the other thing is
this is all full of Python.

We do use Python

to create the firmware for the FPGA,
and all these other areas.

If you wanna find out more about that
go to my talk at Python AU

which was recorded with the very device
I’m talking about.

microphone noise
Oops, sorry!

But as I said, this is going
to include a lot of problems.

The first one is:
People still use VGA!

This kind of makes me sad.

Because VGA is not HDMI.
It was invented in 1987

and it’s an analog signal.

Well, HDMI shares some history
with VGA.

You can’t use the same techniques
for capturing HDMI that you can for VGA.

So why do you still use it?
It’s old and bad!

We developed a VGA expansion board
to effectively allow us to capture

VGA using the same thing.

By ‘developed’ I mean
we have designed, and some exist

but nobody’s actually finished the
firmware to make them work yet.

So, I’d love help there.

There is also another problem:

I want to do this
all open source, as I said.

The HDMI ecosystem has
commercial cores you can buy

and they work reasonably well,
but you have to buy them

and you don’t get the source code to them
or if you do get the source code to them

you can’t share them with other people.

As well, I wanted to be open source
because we wanted to solve

all those problems that people
have when plugging in their laptop

and it not working.

And the commercial cores aren’t designed
to allow us to give the ability

to do that –
solve those problems permanently.

So we create a new implementation!

As anybody who has ever done a
reimplementation or a new implementation

or something, it means that you got
new bugs which I all describe quite a bit.

So this talk could be called

‘Debugging HDMI’ rather than
‘Dissecting HDMI’ because it includes

a lot of information about
how things went wrong.

Ok, that’s kind of the introduction of why
we are here and why I’m talking about this.

So how does HDMI work?

Well, HDMI is actually reasonably old now.

It was created in 2002.

It’s based on the DVI specification.
DVI was created in 1999.

So DVI is 17 years old.

And DVI was designed to replace VGA
and shares a lot of similar history.

HDMI is backwards compatible with DVI
electrically and protocol wise,

but uses a different connector.

This is a HDMI connector.
You’ve probably seen them all before.

If you look closely you see that
there are 19 pins on the HDMI connector.

That’s Pin 1.

So what do all this pins do?

There are five pins
which are used for Ground.

There is one pin which is used for Power,
it gives you 5 volts at 50 milliamps.

This isn’t much.
You can’t do much with 50 milliamps

except maybe some type of adapter,
converter or power a whole microcontroller.

Some Chinese devices try
to draw like an amp from this.

That’s not very good. So that’s another
thing you should watch out for.

There are three high speed data pairs
which transmit the actual video data.

And they share a clock pair.
So that’s these pins here.

And then there are five pins
which are used for low speed data.

So that’s all the pins
on the HDMI connector.

You might have noticed that there was
a whole bunch of different things

I said there. And you need to actually
understand a whole bunch

of different protocols
to understand how HDMI works.

There is bunch of low speed ones
and there is a bunch of high speed ones.

I’m not gonna talk
about all of those protocols

because there is just too many
to go into an hour talk.

The low speed protocol
I’m not gonna talk about is

CEC or Audio Return;
and I’m not gonna talk about any of the

Auxiliary Data protocol
that is high speed, or HDCP.

If you want HDCP go
and look at bunnie’s talk.

It’s much better than mine.
But… or ethernet!

What I will be talking about is
the EDID and DDC protocols,

The 8b/10b encoding of the pixel data

and the 2b/10b encoding
of the control data.

Interesting enough this is actually DVI.

I’m not telling you about HDMI, I’m
really describing to you how DVI works.

Again, many titles.

Starting with the low speed protocol:

EDID or DDC.

I’m gonna use these two terms
interchangeably,

they’ve been so confused now
that they are interchangeable,

in my opinion.

This is something they inherited from VGA.

It was invented and added
to VGA in August 1994.

It was for plug and play of monitors
so that you could plug in your monitor

and your graphics card would just work
rather than requiring you to tell

your graphics card exactly what resolution
and stuff your monitor worked at.

It uses I2C [I squared C]
and a small EEPROM.

These are the pins that it uses.

15 is the Clock pin and
16 is the Data pin,

and then it uses the Ground, and the
5 Volts is used to power that EEPROM.

And in some ways it also uses the 19
because 19 is how you detect

that there is something there
to read from.

It uses I2C.

I2C is a low speed protocol that runs
at either 100 kHz or 400 kHz.

Technically EDID is not I2C,
but it is.

It only supports the 100 kHz version,
though, in theory,

everything on this planet
can be read at 400 kHz.

It is also very well explained elsewhere,
so I’m not going to explain in detail

what I2C is or does,
or how to implement it.

The EEPROM is a 24 series.
It’s found at I2C address 50.

It’s 8 bits in size which gives
you 256 bytes of data.

Again, this EEPROM and how to talk to it
is very well described on internet.

So I’m not gonna to describe it here.
If you’ve used EEPROMs

over I2C it’s likely
you’ve used a 24 series EEPROM.

Probably bigger ones,
256 bytes is pretty small.

So like a 16 bits one,

but EDID supports only
the 8 bits ones.

The kind of interesting part of EDID
is the data structure:

it’s a custom binary format that describes

what the contents of the EEPROM is.

Again, Wikipedia has a really
good description of this.

So I’m not gonna go
into much detail.

But the important thing is that it
describes resolution, frequency

and format for talking to the monitor.

This is really important because
if you try and send

the wrong resolution, frequency or format
the monitor is not gonna understand it.

This is kind of what EDID is used for.

sipping, sound of water bottle

So this is where things
start getting a bit hairy.

Presenters come up to the front and
the first question you see anybody ask is:

What resolution do I use?

And they get a panel like this which
has a bazillion of resolutions selected.

And the thing is, despite your monitor
saying that it supports

many formats they lie.

It turns out that projectors lie
a lot more than normal displays.

I don’t know why they are special.

So this is what a supported format
looks like.

It’s really great.

As well, I care about
capturing the data.

And so I want the things
in the format that is

easy for me to capture.

I also don’t to be scaling
peoples images and text

because scaling looks really bad.
So if someone selects

like a really low resolution and
we scale it up it looks really horrible.

It makes text unreadable; and
presenters are very denounced,

especially at technical conferences,
for using tiny, tiny fonts.

And so we need to use as much
resolution as we can.

How we solve this is we emulate
our own EEPROM in the FPGA

and ignore what the projector
tells us it can do.

We tell the presenter that this is
what we support.

You might notice that it kind of
solves the problem

of what resolution we do.

Offer a single solution…

offer a single option makes it
very hard to choose the wrong one.

That’s good! We solved the problem!

No, we haven’t solved the problem.

We were recording PyCon AU
and we found that

some Mac laptops were
refusing to work.

To understand the cause of this
you need to understand

a little bit about how the world works.

There are two major frequencies
in the world: 50 Hz and 60 Hz.

50 Hz is mainly used
in the “Rest of the World”

and 60 Hz is used in America
and Japan and a few other places

but that’s kind of a very rough thing.

Laptop sold in Australia,
Australia is 50 Hz.

It’s part of the “Rest of the World”.
You’d think the that the laptop

could do 50 Hz. Plus everything
is global these days, right?

I can plug in my power pack
for my laptop in the US or Australia,

like, it should work everywhere right!

No. Sad!

We solved it by claiming
that we were American

and supporting 60 frames per second
rather than 50 frames per second.

So I guess a display
with an American accent.

We deployed this hotfix
on the Friday evening.

And on Saturday all the problems
that we were having on Friday

went away. So this is kind of
the power of a open source solution

and having complete control
over your hardware.

Nowadays we actually offer both 60 and 50

because for display capture
if you’re displaying stuff

at 50 frames per second you’re
probably speaking a lot faster than I am.

It’s really weird, these
128 bytes are really hard

and the number one cause
of why a persons laptop

can’t talk to the projector.

It gets a trophy!

To try and figure out why that is
we created EDID.tv.

It’s supposed to be
a repository of EDID data.

There is a Summer of Code project,
Python/Django/Bootstrap

and an EDID grabber tool that
you can run on your laptop.

I’d love help making this work better.

Hasn’t had much love since
the Summer of Code student made that work.

But it would be really nice to have an
open database of everybody’s EDID data

out there. There are a bunch
of closed ones. I can pay to buy one,

but I’d really love to have an open one.

As well maybe we don’t need
the whole capture solution,

maybe you can just override the EDID.

The C3VOC here actually developed
a version that overrides EDID for VGA.

I have a design which works for HDMI.

It just uses a low cost microprocessor
to pretend to be an EEPROM.

As well, DisplayPort is not HDMI.
Don’t get these two confused,

they are very, very different protocols.

They have an Auxiliary Channel
like EDID and CEC.

I have boards to decode them
here at CCC.

So if you’re interested in that
come and talk to me

because we would really like to do
similar things for DisplayPort.

That is the slow speed data.

Sip from bottle

What about the high speed data?

Each pixel on your screen is

basically three colors
in DVI standard: Red, Green, Blue.

And each one is a byte in size.

Each of the colors is mapped to
a channel on the HDMI connector.

You can kind of see the Red and
the Green and the Blue channels.

Each channel is differential pair.

You get a Plus and a Negative
and a Shield.

And they use twisted pair to try
and reduce the noise reception of these,

because these are quite high speed.

And they have a dedicated Shield to
try and – again – reduce the noise

that is captured.

This is kind of where it gets to
the ‘differential signaling’ part

of the ‘TMDS’ that is
the kind of code name

for the internal protocol that is used
on the high speed data.

They also…
all these channels share a Clock.

That clock is called the Pixel Clock.

But each of these channels
is a serial channel.

It transmits data at 10 bits.

They… every 10 bits – sorry,

every clock cycle there are 10 bits of data
transmitted on each of these channels.

There is a shared clock and
each of the channels is running

at effectively
ten times that shared clock.

This is kind of what
the whole system looks like.

You have your Red, Green, Blue channels.

You take your 8 bits of input data
on each channel

and you convert it to the 10 bits

that we’re going to transmit,
and it goes across the cable

and then we decode it on the other side.

The question is: what does
the 8 bit to 10 bit encoding

look like and how do you understand that.

It is described by this diagram here.
It’s a bit small so I’ll bring it up.

This is what it looks like.
Yes… sure…

…what? This diagram – like –

I’ve spent hours looking at this,
and it is an extremely hard diagram

to decode.
It’s very, very hard to understand.

And it turns out the encoding
protocol is actually quite easy!

It’s three easy steps – approximately.

So I’m going to show you all how
to write an encoder or a decoder.

That diagram is just for the encoder.

They have a similar diagram that
is not the inverse of this for decoding.

Again, almost impossible to read.

The three steps: First we’re
going to do ‘Control’ or ‘Pixel’,

choose which one to do. Then we’re
going to either encode Control data

or encode Pixel data.

A couple of important points
to go through first:

The Input data
– no matter how wide it is –

is converted to 10 bit symbols.

Data goes to symbols.
When we’re talking about them

being transmitted we talk about them
in symbols, when it’s decoded into pixels

we talk about them in data.

As well, things need
to be kept DC-balanced.

I’ve rushed ahead.

The question is: “Why 10 bits?”
Our pixels were 8 bits.

I will explain why
in the Pixel Data section.

But it’s important that all our symbols
are the same size.

We’re always transmitting 10 bits
every clock cycle.

Keeping DC-balanced:

long runs of 1s and 0s are bad.

There are lots of reasons for this.

I tend to think of it like
HDMI isn’t AC coupled

but you can kind of think of it
like AC coupled.

It’s not to recover Clock.

We have a clock pair that is used
to give our Clock signal.

There are lots of lies on internet
that say that the reason

we’re going to keep DC balance
is because of Clock.

But no, that’s not the case.

So what does DC balance mean?

A symbol which has lots of 1s
or lots of 0s

is going to be considered DC-biased

if it has more 1s than 0s.

This is kind of what it’s like:
this symbol here

has lots of 1s and if you add up
all the 1s you can see it’s got

quite a positive bias.
If it was inverse and had lots of 0s

it would have a negative DC bias.

That cause… that DC bias over time
causes us problems.

That are the two important things we have
to keep in mind when looking at the rest.

sound of bottle sip

The first thing we need to figure out is
are we transmitting Control data

or Pixel data.

Turns out that what is happening
in your display is,

we are transmitting something
that’s actually bigger

than what you
see on your screen.

This not the scale. The Control data
periods are much, much smaller.

The Control data is in orange
and the Pixel data is in purple-pink.

So why does this exist?
It exists because of old CRT monitors.

And for those in the audience
who where kind of born after CRT monitors,

this is what they look like.

The way they work is,
they have an electron beam

that scans across,
highlighting the phosphorus.

This electron beam can’t just be…
get back to other side of the screen

straight away, or get back to the top of
the screen. And so these periods

where we are transmitting Control data
was to allow the electron beam

to get back to the location
where it needed to start

transmitting the next set of data.

That’s why it exists.
Why do we care?

Because the encoding schemes
for Control and Pixel data

are actually quite different.

This is the main difference.
I’m going to come back to this slide

a bit later. But again, the
important thing to see here is

that despite the encoding scheme
being quite different

the output is 10 bits in size.

That first step – choosing whether
it’s Pixel or Control data –

is described by this bit of the diagram.
You might notice that’s

not the first thing in the diagram.

How do you convert Control data
to Control symbols?

First we need to know what
Control data is. There are two bits,

there is the HSync and the VSync signal.

They provide basically
the horizontal and vertical pixel sizes.

They are kind of left over from VGA.
We don’t actually need them

in HDMI or DVI to know
where the edges are

because we can tell the difference
between Control and Pixel data.

But they kind of still exist
because of backwards compatibility.

This means that we have two bits of data
that we need to convert to 10 bits of data.

So, a 2b/10b scheme.

How they do it is they just hand-picked
four symbols that were going to be

these Control data symbols.

These are the four symbols. There’s
some interesting properties with them.

They are chosen to be DC-balanced.
They roughly have the same number

of 0s and 1s. So we don’t have to worry about
the DC bias with these symbols very much.

They are also chosen to have
seven or more transitions from 0 to 1

in them. This number of transitions

is used to understand
the phase relationship

of the different channels.
So if you remember this diagram,

we have a cable going between
the transmitter and the transceiver.

These are, again, very high speed signals.

And even if the transmitter was
transmitting everything at the same time,

the cable isn’t ideal and might
delay some of the symbols.

The bits on one channel
[might take] longer than others.

By having lots of these transmissions
we can actually find

the phase relationship between
each of the channels and then

recover the data. And so
that’s why these Control symbols

have a large number
of transitions in them.

More on that later when we get to the
implementation. And I’m running out’ time.

This part of the diagram is the
Control data encoding.

sip from bottle

What about Pixel data
and the Pixel symbols?

Again, in DVI each channel
of the Pixel is 8 bits.

And the encoding scheme is described
by the rest of the diagram.

But again, it’s actually
really, really simple.

This encoding scheme is called 8b/10b,

because it takes 8 bits
converting it to 10 bits.

However, there is a huge danger
here because IBM also invented

the 8b/10b scheme
that is used in everything.

This is used in DisplayPort, it’s used
in PCI Express, it’s used in SATA,

it’s used in pretty much everything
on the planet.

This is not the encoding TDMS uses.

You can lose a lot of time
trying to map this diagram

to the IBM coding scheme,
and going these are not the same.

That is because they’re not the same.
This is a totally different coding scheme.

Encoding Pixel data is a two-step process.
I did say it was three-ish steps

to do this.
The first step is we want to reduce

the transitions in the data.

How do we do this? –
Sorry, why do we do this?

Because this, again, is
a high speed channel.

We want to reduce the cross-talk
between the lanes.

They are actually quite close
to each other.

So by reducing the number
of transitions we can reduce

the probability that the signal propagates

from one channel to the next.
And how we do it?

We’re gonna choose one
of two encoding schemes.

An XOR encoding scheme
or an XNOR encoding scheme.

How do we do the XOR encoding scheme?
It’s actually pretty simple.

We set the Encoded Bit
same as the first Data Bit

and then the next Encoded Bit
is the first Encoded Bit

XORed with the Data bit.

And then we just repeat until
we’ve done the 8 bits.

So this is how we do the XOR encoding.

The XNOR encoding is the same process,
except instead of using XOR

it uses XNOR.

How do we choose
which one of these to use?

If the Input Data byte
has fewer than four 1s

we use the XOR. If it has more
than four 1s we use the XNOR.

And then there’s a tie-break (?)
if you have even.

The important thing here is that this
method is determined by the Data byte only.

There is no hidden state here
or continuous change.

Every pixel has a one-to-one
mapping to an encoding.

Then we append a bit
on the end that indicates

whether we chose XOR,
XNOR encoding of that data.

And so that converts
our 8 bits Input Pixels

to 9 bits of encoded data, effectively
our 8-bit encoded sequence

and then one bit to indicate whether
we chose XOR, or XNOR encoding

for that Data bit. So that’s it there.

This encoding is actually very good
at reducing transitions.

On average, we had roughly
eight transitions previously,

now we have roughly three-ish,
so it’s pretty cool.

I have no idea how they figured this out.

I’m assuming some very smart
mathematicians where involved

because discovering this is beyond me.

And that describes the top part
of this process.

sounds of scratching nose and beard

This is where, in the TMDS, the
Transition Minimization comes from,

that step there in the encoding process.

But there is still one more step.

We need to keep the channel
DC-balanced, as I explained earlier.

How can we do that? Because
not all pixels are guaranteed to be

at zero DC bias
like the Control symbols are.

We do it by keeping a running count
of the DC bias we have,

and then, if we have a positive DC bias

and the symbol is also
positively biased, we invert it.

Or, if we have a negative DC bias
and the symbol has a negative DC bias,

we invert it.
And the reason we do this is

because when we invert a symbol we
convert all the 1s to 0s which means

a negative DC bias
becomes a positive DC bias.

As I said, we chose – because we are already
negative and the thing was negative –

we convert it to plus. It means we’re
going to drive the running DC bias value

back towards zero.
We might overshoot, but the next stage

we’ll keep trying to oscillate up and
down, and on average over time

we keep a DC bias of zero.

And as I said. Then, to indicate
whether or not we inverted

or kept… the…
straight through we inverted,

we add another bit on the end.
So that’s how get our 10 bit

encoding scheme.
We have the 8 bits of encoded data,

then one bit indicating whether or not
it used XOR/XNOR encoding,

and then one bit to indicate whether
or not we inverted the symbol.

That describes this bottom part
of the chart.

Now you can see partly
why this chart is kind of confusing.

It’s no way in what I think
of a what’s a logical diagram.

This might be how you implement it
in hardware if you really understand

the protocol, but not a very good diagram
for explaining what’s going on. And…

sip from bottle

As you see it’s actually pretty simple.

In summary this is
the interesting information

about the two different encoding schemes.

Because we minimized
the transitions in the Pixel data

we can actually tell
Control and Pixel data apart

by looking at how many transitions
are in the symbol.

If it has six or more transitions
it must be a Control symbol.

If it has four or less
it must be a Pixel symbol.

You now know
how to encode TDMS data

and how to decode TDMS data

because if you want to decode
you just do the process backwards.

Congratulations!
How do you actually implement this?

You can just write the XOR logic

and a little counter
that keeps track of the DC bias

and all that type of thing
in FPGA.

I’m not going to describe that
because I don’t have much time.

But if you followed the process
that I have given you

it should be pretty easy.

But… and this is what we use currently.

You could actually use a lookup table.
What we are doing is

converting 8 bits of data
to 10 bits of data.

That is a lookup table process,
pretty easy.

FPGAs are really good at
doing ‘lookup table’-type processes,

and it also allows you then
to extend this system

to those other protocols
like the 4b/10b that is used

for the Auxiliary data.

So we are looking at that in the future.
It uses a few more resources

but it’s a lot more powerful.

This is kind of what your encoder
will look like, and your decoder.

It’s quite simple,
it takes in your 10 bits of data

and outputs either
your 8 bits of Pixel data

or your 2 bits of Control data
and the data type.

This is kind of what if you went
into our design and looked at it

at high level, in the schematic,

you’d probably see a block
that looks like this.

The encoder is slightly more complicated
because you also have the DC bias count

that you have to keep track of.
But, again,

the data goes in
and the data comes out.

That’s simple, right?

This kind of extends to Auxiliary data,
or if you get an error,

like if you…
There are 124 symbols

that you can have in 10 bits of data.

Not all of them are valid.
So if you get one of these invalid symbols

you know you have an error.

However, things happen quite quickly

when you times them by ten.
And so our Pixel Clock

for 640x480 is 25 MHz.
When you times that by ten

you get 250 MBits per channel.
When you’re doing 720p

you’re doing 750 MBits per channel.

And 1080p is at 1500 MBits per channel.

An FPGA is fast, but
they’re not really that fast

at a range that I can afford to buy.
I’m sure the military has ones

that go this fast, but
I’m not as rich as them.

But they do include a nice hack
to solve this.

They are called SerDes.
They basically turn parallel data

into serial data.

This is what the boxes look like.

You give them your TDMS parallel data

and they convert it to
high speed serial data for you.

They are a little bit fiddly to use
and your best option is to go and find

a person who has already configured this
for your FPGA

and follow what they do.

“Hamster” – Mike “Hamster” Field – has
a really good documentation on

how to use these in a Spartan6.
These are also unique to your FPGA,

so different FPGAs are going to have
different control schemes.

But if you are using a Spartan6

then go and look up what
Mike “Hamster” Field is

doing for configuring these.

Remember how I said,
our system has a serial console.

Because we have that system
we can actually delve quite deep

into what’s happening
internally in the system.

sip from bottle

And print it out.
This is debugging from one of our systems.

You can see…

The first thing is the phase relationship
between each of the channels.

The next one is whether
we’re getting valid data

on each of the channels and then
we’ve got the error rate for that channel,

whether all channels synchronized,
and then some resolution information.

You can see that this has got
a 74 MHz Pixel Clock.

There are three columns because
there is Red, Green and Blue channels.

This give us some very interesting
debugging capabilities.

If you plug in a cable
and you’re getting errors

on the Blue channel and nowhere else

it’s highly likely there’s
something wrong with that cable.

This is a very powerful tool
that allows us to figure out

what’s going wrong in a system.

It’s something you can’t really get
with the commercial versions of this.

But what about errors?
Everything I’m talking about now

is a little bit experimental,
we haven’t actually implemented this.

But it’s some ideas about
what we can do because we now

have complete control of our decoder.

As I said, there’s 124 possible choices
for 10 bit symbols,

of which 460 are valid Pixel symbols,

4 are valid Control symbols
and 560 symbols

should never ever be seen no matter what.

That’s like 56% of our space
that should never be seen.

But it’s actually better than that!
We know because of the running DC bias

that there are 256 valid Pixel symbols

at any one point. You can’t have the…
if you’ve got a negative DC bias

you can’t have a Pixel symbol
which continues to drive you negative.

Actually, 74% of our space at any time

is not allowed to exist.

This means that a huge number
of the invalid symbols

are only near one other valid symbol.

And so we can actually correct them!
We can go: “This symbol must have been

this other symbol,
because it’s not a valid symbol,

it must be a bit error
from this other symbol.”

So we can correct these errors.
This is quite cool.

We can correct about 70% of

single bit flip errors in Pixel data.

But sadly there is some that we can’t.

But we can detect that we got
a invalid Pixel data.

So the fact that there is an error
is important.

In this case we’ve got two pixels
that we received correctly

and we got a pixel that we know
is a invalid value

and then two more pixels
that we received correctly.

You can imagine this is a Blue channel,

so the first ones were not very blue.

Then there’s the decoded value for this is

very, very blue, like very light blue
and then some other ones.

This looks really bad, right?

This was probably a whole blue block.

One pixel difference
of that big, that size,

is probably not a valid value,

and so we can cover them up!

We can go…
the two pixels on either side

and average them and fix that pixel.

This allow us to correct a whole bunch
more of errors that are occurring.

And as we’re about to take this data

and run it through a JPEG encoder

this doesn’t actually affect
the quality of the output

all that much and allows to fix
things that would otherwise

be giant glaring glitches in the output.

That’s some interesting information about

how you do TDMS decoding
and how we can fix some errors.

The thing is, we can do it
even better than this

because it’s an open source project.

Maybe you have some idea
about how we can improve

the SerDes performance.
Maybe you have an idea about

how to do TDMS decoding on
much lower power devices

than we use. It’s open source!
You can look at the code

and you can improve it.
And we would love you to do it!

The thing is that I have a lot of hardware
but not much time.

If you have lots of time
and not much hardware,

I think I can solve this problem.

These are links to the HDMI2USB project

and the TimVideos project;
and all our code, our hardware,

everything is on GitHub
under open source licenses.

And here is some bonus screen shots that
I wasn’t able to fit in other locations.

You can see these small errors.

That one was kind of a big error.

This is what happens when
your DDR memory is slightly broken.

Yeah…
but – yeah!

And that is my talk!

applause

Herald: Excellent!
Thank you very much, mithro.

As you’ve noticed, we have a couple of
microphones standing around in the room.

If you have any questions for mithro
please line up behind the microphones

and I will allow you to ask the questions.
We have question from the Internet!?

Signal Angel: Yes, thank you!
Do you know if normal monitors

do similar error recovery or hiding?

Tim: I know of no commercial
implementation that does

any type of error correction.
The solution for the commercial guys

is to effectively never get errors.

They can do that because

they don’t have to deal with
the angry speakers on the ground

going wild as my slides look weird.

And, as well, they are probably working
with better quality hardware

than we are using. We’re trying
to make things as cheap as possible.

And so we are pushing the boundaries
of a lot of the devices we are using.

So we are more likely to get
errors than they are.

Herald: We have quite a lot of questions.
Remember – questions, not comments!

Microphone number 1, please!

rustling sound from audience
coughing

Tim: Yes!

unrecognizable question from audience

Sorry, I don’t quite understand
what’s going on! chuckles

Herald: Do we have a translation?

Voice from audience: Audio Angel!

Tim: Audio problem?

Herald speaks to person
in front of stage in German

Tim: I’ll be around afterwards,
If you want to chat to me, ahm…

Herald: And we might do that… ah…
write you on the computer afterwards.

Second question from
microphone number 3, please!

Question: Hello? Ah, yes. Can you
determine the quality of a HDMI cable,

e.g. by measuring bit error rate
of each three pairs

and also some jitter on the clock,
and that kind of…?

Tim: Yes we can!

The quality of a HDMI cable should be
they’re zero bit errors.

So anything that has non-zero bit errors
we chop up and throw away.

This gets interesting
when you have very long cables.

We can actually see that
the longer the cable is

the harder for them
to keep zero bit errors.

So yes, we can kind of judge
the quality of the cable.

But it’s also hard because

it depends on what the sender is doing.

If the sender is of a lower quality

and the cable is low quality
you might get bit errors.

But if the sender is of a high quality

and the cable is of a lower quality

they might cancel each other out
and still be fine.

We can’t just go: “This is a good cable”

because we don’t actually have
any control over our…

how powerful our sender is on this device.

If we could kind of turn down the sender

and see where things start going wrong
that would be pretty cool.

If anybody wants to
look at building such a device

I’d love to help you do that.

Herald: We have another question
from microphone number 5.

Question: Your hardware,
the HDMI2USB hardware…

Tim: Yes!
Question: Is it available for simply ordering

or has it to be solder soldered by hand,
or…

Tim: You can not solder this board by
hand unless you are much, much better

than I am. It uses Ball Grid Array parts
because it’s an FPGA.

This is one here.
You can buy them.

We’re working with a manufacturer in India
who builds them for us.

We work with them,
and it was pretty awesome.

We’re also working on new hardware.
I’ve got a whole bunch

of FPGA hardware here
that you can come and have a look at

and I’ll probably move it out
into the hallway afterwards.

Again, if you’re interested
in the hardware and you have a use case,

chat to me!
Because I like to solve problems

of people who are not having hardware
and my employer pays me too much.

So I get to use my discretion refunds

for helping out people
doing open source stuff.

applause

Herald: We have at least four more
questions. Microphone number 2, please!

Question: Do you think it would be
possible to get an 1080p image

out of the open source
hardware board you use?

Tim: Yes, I do, but it requires
us to do some hard work

that we haven’t had time to do yet.

And for us 720p at 60 frames
per second is good enough.

The USB connection

is limited in bandwidth because
we don’t have an H.264 encoder,

we only have MJPEG. If somebody wants
to write us a open source, say, WebM

rather than H.264 encoder

that might start become more interesting.
We also have ethernet, Gigabit ethernet,

on this board. It should be pretty ease
to stream the data out the ethernet.

I, again, need help.

The ethernet controller works.
We can telnet into the board

and control it via Telnet.
We just need somebody to

actually connect the data,
the high speed data side up.

We use it for debugging and stuff.

Mike “Hamster” Field again,
really big thank you to him,

he is an amazing designer,

he built 1080p60 that is
a little bit out-of-spec

but actually works really well on hardware

that is almost identical to ours.
He also did the DisplayPort,

like a 4k-DisplayPort which
we can do on our board.

If you only need one or two
of the 1080p things

DisplayPort connectors can be
converted to HDMI quite easily

and you can do that on them.

Yes, I think that’s possible,
but again:

open source … hobbyist …
need developers …

Herald: We’ll take one question
from the internet!

Signal Angel: Thank you. Have you
considered JPEG2000?

Tim: No, I’ve not. And the main reason
is that I want to be a webcam.

I want to pretend to be a webcam. The UVC
standard, which is the USB webcam standard,

does not support JPEG2000.

There’s no reason we couldn’t support
JPEG2000 when connected to Linux,

like we could fix the Linux driver
to add JPEG2000 support.

Again, I don’t know if there’s any good
open source FPGA implementations

of JPEG2000? So, that’s also a blocker.

But if you’re interested in helping out
– come and talk to me.

As I said, I would very much love

to chat to you and solve
the problems you’re having

with getting-going.
As well, we have t-shirts.

I’m wearing a t-shirt that we have, and
I will send everybody who contributes

a t-shirt. Whether that’s fixing our
website, helping on documentation,

helping people on IRC
getting setup, anything.

You don’t need to be an expert
on FPGA stuff to help out.

And we also are working on
a little project to run MicroPython

on FPGAs. So if you’re really into Python
and you like MicroPython

I would love to help you help us do that.

It’s kind of working. We just need more
powerful (?) support. So.

Herald: We have two more questions from
microphone number 1.

Question: So, is there some sort of
dedicated processor on that board,

or do you use like a Microblaze
in the FPGA?

Tim: We use an open source soft core.
One of three.

We can change which soft core
we’re using with a command line flag.

We’re using either the LatticeMico32

which is produced
by Lattice Semiconductor.

We can use the OpenRISC-1000

or we can use a RISC-V processor.

We generally default to LM32
because it’s the most performance

for least FPGA resource trade-off.

But if you like RISC-V
or OpenRISC-1000 better

for some reason, say, you want
to run Linux on our soft core,

then you can do that. With a one line
command line change, yeah!

We’re looking at adding
J-Core support in early next year.

J-Core is quite big, though,
compared to LM32. So,

it probably won’t fit on some
of the very small devices.

Question: So it’s a Lattice FPGA?

Tim: It’s a Spartan6 FPGA. And our new
boards will probably be Artix-7

But we’re still in the process
of making them exist yet.

Question: Thanks.
Tim: I’ve also been working with

bunnie’s NeTV2, porting
our firmware to that,

which has been really awesome.
He’s doing some cool work there,

and he’s kind of inspired this whole
development by showing that,

yes, you could do this,
and you shouldn’t be scared of it.

Herald: Good, one more question
from microphone number 1.

Question: Yes. Do you have any plans for
incorporating HD-SDI into your platform?

Tim: Yes and no!
We have plans and ideas

that we could do it

but HD-SDI

and all of the SDI protocols are
much harder for the consumer,

generally to access, and we want
to drive the costs of this down

to as low as it can go. And…

HDMI is a consumer electronic thing.
You get it on everything.

You get it on your
like five-buck Raspberry Pi.

HDMI is probably
a really good solution for this.

We haven’t developed any SDI cores
or anything like that,

so I can’t tell you like
that we’re doing anything there

but if somebody’s interested, again,
I like to remove roll (?) blocks and

we would love to have people work on that.

Herald: We have one more question from
the internet and we have two minutes left.

Signal Angel: OK, thank you. The question
is not related to HDMI but to FPGAs.

FPGAs are programmed in a high level
language like VERILOG or…

after simulation you compile. So every
vendor has created his own compiler

for its own hardware. Are you aware of
a move to open source compilers

or to independent hardware? And do you see
a benefit in open source FPGA compilers?

Tim: Yes! If anybody knows

about FPGAs you know
they use proprietary compilers.

And these proprietary compilers
are terrible.

I’m a software engineer.
If I find a bug in gcc

I can fix the bug. I’ve got those skills,
and I can move forward or at least

figure out why the hell the bug occurred.

That is not the case with FPGA compilers.
The FPGA compiler we use

is non-deterministic. You can give it
the same source code and it produces

different output. I’d love somebody
to reverse-engineer why that occurs

because I’ve removed all the randomness
from random sources from it

and it still manages to do it!
I’m really impressed. So,

Clifford has done an open source
FPGA tool chain

for the Lattice iCEstick things.

He said he’s gonna work
on the Actrix7 FPGAs.

Please donate to him and help him.
I would… like…

if that exists I owe people who…
like a bazillion of beers, because

the sooner I can get off proprietary
tool chains the happier I will be,

and it will make my hobby so much nicer.
So, please help him!

Herald: And do give Tim
a big round of applause!

applause

postroll music

subtitles created by c3subtitles.de
in the year 2017. Join, and help us!