0:00:00.000,0:00:14.309
33C3 preroll music
0:00:14.309,0:00:17.380
Herald-Angel: Tim ‘mithro’ Ansell[br]has come all the way from Australia
0:00:17.380,0:00:24.510
to talk to us about ‘Dissecting HDMI’[br]and developing an open FPGA-based
0:00:24.510,0:00:29.930
capture hardware for sharing[br]talks outside of the room.
0:00:29.930,0:00:34.680
And he will be explaining how[br]to dissect it. And I’m looking forward
0:00:34.680,0:00:38.620
to hearing the talk in a second.[br]And so please give Tim
0:00:38.620,0:00:41.330
a warm round of applause[br]again. Thank you!
0:00:41.330,0:00:50.520
applause
0:00:50.520,0:00:56.149
Tim Ansell: Okay, hi, I’m Tim, and[br]in theory, if my slides change
0:00:56.149,0:01:02.550
you would see that.
0:01:02.550,0:01:08.540
And I kind of have too many projects.[br]And I’m gonna be discussing one of them.
0:01:08.540,0:01:11.890
This is another project that[br]I gave a lightning talk on earlier.
0:01:11.890,0:01:18.720
If you didn’t see it, it’s an ARM[br]microcontroller that goes in your USB port.
0:01:18.720,0:01:22.000
People wanted to know[br]when to hack on it.
0:01:22.000,0:01:25.520
Tomorrow at 2 PM, apparently.
0:01:25.520,0:01:28.990
So, the first that I want to say is[br]I’m a software developer.
0:01:28.990,0:01:33.340
I’m not a hardware designer,[br]I’m not an FPGA developer,
0:01:33.340,0:01:37.150
I’m not a professional in any of that.
0:01:37.150,0:01:42.920
I develop software for full time.[br]So this is my hobby.
0:01:42.920,0:01:50.029
As well, this information comes from[br]a couple of projects that I started
0:01:50.029,0:01:55.260
but a lot of other people did the majority[br]of the work and I’m just telling you
0:01:55.260,0:02:00.180
about it because they are too shy[br]to come up and talk about it themselves.
0:02:00.180,0:02:05.430
So a big thank you to all these people[br]who have helped me in various ways
0:02:05.430,0:02:09.788
regarding this. And theses slides –
0:02:09.788,0:02:13.700
any of the blue things are links, so[br]if your playing it along at home
0:02:13.700,0:02:18.840
you can get to them by that URL[br]and click on these things.
0:02:18.840,0:02:21.700
And there is probably other people I’ve[br]forgotten, who are not on this list.
0:02:21.700,0:02:24.530
I’m very sorry.
0:02:24.530,0:02:31.980
So this title of this talk could be called[br]“Software guy tries hardware and complains”.
0:02:31.980,0:02:36.640
I’ve had a really hard time figuring out[br]what to call this talk.
0:02:36.640,0:02:41.210
And you’ll see some other attempts[br]of naming this talk better.
0:02:41.210,0:02:48.379
So a bit of history.[br]How do I end up doing HDMI stuff?
0:02:48.379,0:02:55.840
So TimVideos is a group of projects[br]which are trying to make it easy
0:02:55.840,0:03:02.780
to record and live-stream user groups[br]and conferences like this event.
0:03:02.780,0:03:07.400
However, we want to do it without[br]needing the awesome team
0:03:07.400,0:03:09.750
that is doing the recording here.
0:03:09.750,0:03:12.950
These guys are really, really organized[br]and professional.
0:03:12.950,0:03:18.710
We want to do it where people have[br]no experience at all with AV,
0:03:18.710,0:03:22.110
and just can make it happen.
0:03:22.110,0:03:28.930
And so this is how you record[br]a conference or user group.
0:03:28.930,0:03:31.750
I’m gonna be talking about[br]these two things here,
0:03:31.750,0:03:35.519
the HDMI2USB devices that we created.
0:03:35.519,0:03:43.239
They are used in our setup both for[br]camera capture and for capture of slides.
0:03:43.239,0:03:48.680
And so HDMI2USB is FOSS hardware[br]for doing HDMI capture
0:03:48.680,0:03:54.880
and actually has a bit of history[br]with the CCC
0:03:54.880,0:04:00.700
because it was inspired by[br]a speaker who spoke here.
0:04:00.700,0:04:06.139
Bunnie spoke on his NeTV board
0:04:06.139,0:04:08.830
which was an FPGA Man-in-the-Middle attack
0:04:08.830,0:04:14.239
on HDCP-secured links. His talk[br]is really awesome. It’s gonna be…
0:04:14.239,0:04:17.488
that talk is way more technical[br]than mine and gives you
0:04:17.488,0:04:22.460
some really awesome details about the[br]cool things he did to make that work.
0:04:22.460,0:04:26.990
Mine is much more basic. You don’t[br]need much experience with HDMI
0:04:26.990,0:04:32.969
to follow my talk.[br]Our device works like his does,
0:04:32.969,0:04:38.740
except his was deliberately designed[br]to not allow capture,
0:04:38.740,0:04:41.669
our design allows capture.[br]It effectively man-in-the-middles
0:04:41.669,0:04:46.949
the presenters projector,[br]between the presenters laptop
0:04:46.949,0:04:54.859
and the projector, and provides[br]a high-quality capture out the USB2 port.
0:04:54.859,0:04:58.680
I use an FPGA to do that.[br]This is because
0:04:58.680,0:05:03.169
using FPGA makes hardware problems[br]software problems, and as I said
0:05:03.169,0:05:09.110
I’m a software developer, I prefer[br]software problems to hardware problems.
0:05:09.110,0:05:13.689
And the way it kind of works is[br]it appears as a UVC webcam so that
0:05:13.689,0:05:17.589
you can use it with Skype or Hangouts[br]or any of those things without needing
0:05:17.589,0:05:23.650
any drivers on sensible operating[br]systems like MacOS and Linux.
0:05:23.650,0:05:27.620
On Windows you need a driver that[br]tells it to use the internal driver.
0:05:27.620,0:05:30.309
It’s kind of weird.[br]And also a serial port
0:05:30.309,0:05:34.529
because we have the ability to switch[br]which input goes to which output.
0:05:34.529,0:05:38.610
It’s kind of like a matrix.
0:05:38.610,0:05:42.690
And so this the open source[br]hardware we designed.
0:05:42.690,0:05:49.159
It’s in KiCAD, you can find it on Github.
0:05:49.159,0:05:52.320
I’m quite proud of it.[br]It’s quite a good little kit.
0:05:52.320,0:05:58.049
We don’t use all the features of it yet,[br]but it’s pretty awesome.
0:05:58.049,0:06:00.520
And it’s in use!
0:06:00.520,0:06:05.740
We used this technology to capture[br]at a bunch of conferences:
0:06:05.740,0:06:09.309
PyCon in Australia,[br]linux.conf.au in Australia
0:06:09.309,0:06:11.840
– as I said, I’m Australian.
0:06:11.840,0:06:15.059
DebConf, though, are not Australian.
0:06:15.059,0:06:22.789
They used it in – sorry –[br]in South Africa, I think.
0:06:22.789,0:06:24.969
And there are a whole bunch of[br]other people around the world
0:06:24.969,0:06:27.639
who are using this,[br]which is pretty awesome.
0:06:27.639,0:06:31.139
The main reason I wanted it to be[br]open source was so that
0:06:31.139,0:06:37.229
other people could use them[br]and learn from them and fix problems,
0:06:37.229,0:06:41.730
because there are a lot of problems[br]we’ve run into.
0:06:41.730,0:06:44.590
And the other thing is[br]this is all full of Python.
0:06:44.590,0:06:49.979
We do use Python
0:06:49.979,0:06:54.500
to create the firmware for the FPGA,[br]and all these other areas.
0:06:54.500,0:06:58.399
If you wanna find out more about that[br]go to my talk at Python AU
0:06:58.399,0:07:02.679
which was recorded with the very device[br]I’m talking about.
0:07:02.679,0:07:05.419
microphone noise[br]Oops, sorry!
0:07:05.419,0:07:11.759
But as I said, this is going[br]to include a lot of problems.
0:07:11.759,0:07:15.789
The first one is:[br]People still use VGA!
0:07:15.789,0:07:19.429
This kind of makes me sad.
0:07:19.429,0:07:23.849
Because VGA is not HDMI.[br]It was invented in 1987
0:07:23.849,0:07:25.960
and it’s an analog signal.
0:07:25.960,0:07:29.709
Well, HDMI shares some history[br]with VGA.
0:07:29.709,0:07:34.669
You can’t use the same techniques[br]for capturing HDMI that you can for VGA.
0:07:34.669,0:07:40.729
So why do you still use it?[br]It’s old and bad!
0:07:40.729,0:07:46.279
We developed a VGA expansion board[br]to effectively allow us to capture
0:07:46.279,0:07:49.009
VGA using the same thing.
0:07:49.009,0:07:52.699
By ‘developed’ I mean[br]we have designed, and some exist
0:07:52.699,0:07:55.849
but nobody’s actually finished the[br]firmware to make them work yet.
0:07:55.849,0:07:59.530
So, I’d love help there.
0:07:59.530,0:08:03.889
There is also another problem:
0:08:03.889,0:08:08.050
I want to do this[br]all open source, as I said.
0:08:08.050,0:08:12.189
The HDMI ecosystem has[br]commercial cores you can buy
0:08:12.189,0:08:16.089
and they work reasonably well,[br]but you have to buy them
0:08:16.089,0:08:19.550
and you don’t get the source code to them[br]or if you do get the source code to them
0:08:19.550,0:08:23.349
you can’t share them with other people.
0:08:23.349,0:08:27.389
As well, I wanted to be open source[br]because we wanted to solve
0:08:27.389,0:08:31.479
all those problems that people[br]have when plugging in their laptop
0:08:31.479,0:08:34.320
and it not working.
0:08:34.320,0:08:38.969
And the commercial cores aren’t designed[br]to allow us to give the ability
0:08:38.969,0:08:46.550
to do that –[br]solve those problems permanently.
0:08:46.550,0:08:48.500
So we create a new implementation!
0:08:48.500,0:08:52.129
As anybody who has ever done a[br]reimplementation or a new implementation
0:08:52.129,0:08:59.430
or something, it means that you got[br]new bugs which I all describe quite a bit.
0:08:59.430,0:09:01.350
So this talk could be called
0:09:01.350,0:09:06.399
‘Debugging HDMI’ rather than[br]‘Dissecting HDMI’ because it includes
0:09:06.399,0:09:11.590
a lot of information about[br]how things went wrong.
0:09:11.590,0:09:16.560
Ok, that’s kind of the introduction of why[br]we are here and why I’m talking about this.
0:09:16.560,0:09:22.540
So how does HDMI work?
0:09:22.540,0:09:26.480
Well, HDMI is actually reasonably old now.
0:09:26.480,0:09:30.700
It was created in 2002.
0:09:30.700,0:09:37.290
It’s based on the DVI specification.[br]DVI was created in 1999.
0:09:37.290,0:09:41.279
So DVI is 17 years old.
0:09:41.279,0:09:49.070
And DVI was designed to replace VGA[br]and shares a lot of similar history.
0:09:49.070,0:09:55.980
HDMI is backwards compatible with DVI[br]electrically and protocol wise,
0:09:55.980,0:09:58.529
but uses a different connector.
0:09:58.529,0:10:02.750
This is a HDMI connector.[br]You’ve probably seen them all before.
0:10:02.750,0:10:10.220
If you look closely you see that[br]there are 19 pins on the HDMI connector.
0:10:10.220,0:10:13.300
That’s Pin 1.
0:10:13.300,0:10:15.350
So what do all this pins do?
0:10:15.350,0:10:18.470
There are five pins[br]which are used for Ground.
0:10:18.470,0:10:24.829
There is one pin which is used for Power,[br]it gives you 5 volts at 50 milliamps.
0:10:24.829,0:10:28.410
This isn’t much.[br]You can’t do much with 50 milliamps
0:10:28.410,0:10:35.779
except maybe some type of adapter,[br]converter or power a whole microcontroller.
0:10:35.779,0:10:39.290
Some Chinese devices try[br]to draw like an amp from this.
0:10:39.290,0:10:45.370
That’s not very good. So that’s another[br]thing you should watch out for.
0:10:45.370,0:10:51.519
There are three high speed data pairs[br]which transmit the actual video data.
0:10:51.519,0:10:57.730
And they share a clock pair.[br]So that’s these pins here.
0:10:57.730,0:11:02.369
And then there are five pins[br]which are used for low speed data.
0:11:02.369,0:11:06.740
So that’s all the pins[br]on the HDMI connector.
0:11:06.740,0:11:12.740
You might have noticed that there was[br]a whole bunch of different things
0:11:12.740,0:11:16.629
I said there. And you need to actually[br]understand a whole bunch
0:11:16.629,0:11:21.510
of different protocols[br]to understand how HDMI works.
0:11:21.510,0:11:27.710
There is bunch of low speed ones[br]and there is a bunch of high speed ones.
0:11:27.710,0:11:31.910
I’m not gonna talk[br]about all of those protocols
0:11:31.910,0:11:35.810
because there is just too many[br]to go into an hour talk.
0:11:35.810,0:11:38.390
The low speed protocol[br]I’m not gonna talk about is
0:11:38.390,0:11:42.339
CEC or Audio Return;[br]and I’m not gonna talk about any of the
0:11:42.339,0:11:48.010
Auxiliary Data protocol[br]that is high speed, or HDCP.
0:11:48.010,0:11:50.860
If you want HDCP go[br]and look at bunnie’s talk.
0:11:50.860,0:11:56.320
It’s much better than mine.[br]But… or ethernet!
0:11:56.320,0:12:02.460
What I will be talking about is[br]the EDID and DDC protocols,
0:12:02.460,0:12:06.290
The 8b/10b encoding of the pixel data
0:12:06.290,0:12:10.140
and the 2b/10b encoding[br]of the control data.
0:12:10.140,0:12:14.010
Interesting enough this is actually DVI.
0:12:14.010,0:12:20.740
I’m not telling you about HDMI, I’m[br]really describing to you how DVI works.
0:12:20.740,0:12:26.190
Again, many titles.
0:12:26.190,0:12:29.690
Starting with the low speed protocol:
0:12:29.690,0:12:33.009
EDID or DDC.
0:12:33.009,0:12:37.789
I’m gonna use these two terms[br]interchangeably,
0:12:37.789,0:12:40.639
they’ve been so confused now[br]that they are interchangeable,
0:12:40.639,0:12:43.600
in my opinion.
0:12:43.600,0:12:47.480
This is something they inherited from VGA.
0:12:47.480,0:12:52.929
It was invented and added[br]to VGA in August 1994.
0:12:52.929,0:12:56.579
It was for plug and play of monitors[br]so that you could plug in your monitor
0:12:56.579,0:13:02.430
and your graphics card would just work[br]rather than requiring you to tell
0:13:02.430,0:13:08.050
your graphics card exactly what resolution[br]and stuff your monitor worked at.
0:13:08.050,0:13:12.879
It uses I2C [I squared C][br]and a small EEPROM.
0:13:12.879,0:13:16.290
These are the pins that it uses.
0:13:16.290,0:13:22.620
15 is the Clock pin and[br]16 is the Data pin,
0:13:22.620,0:13:28.100
and then it uses the Ground, and the[br]5 Volts is used to power that EEPROM.
0:13:28.100,0:13:32.790
And in some ways it also uses the 19[br]because 19 is how you detect
0:13:32.790,0:13:37.640
that there is something there[br]to read from.
0:13:37.640,0:13:40.569
It uses I2C.
0:13:40.569,0:13:46.190
I2C is a low speed protocol that runs[br]at either 100 kHz or 400 kHz.
0:13:46.190,0:13:50.860
Technically EDID is not I2C,[br]but it is.
0:13:50.860,0:13:55.850
It only supports the 100 kHz version,[br]though, in theory,
0:13:55.850,0:13:58.780
everything on this planet[br]can be read at 400 kHz.
0:13:58.780,0:14:03.029
It is also very well explained elsewhere,[br]so I’m not going to explain in detail
0:14:03.029,0:14:09.089
what I2C is or does,[br]or how to implement it.
0:14:09.089,0:14:16.470
The EEPROM is a 24 series.[br]It’s found at I2C address 50.
0:14:16.470,0:14:21.680
It’s 8 bits in size which gives[br]you 256 bytes of data.
0:14:21.680,0:14:28.240
Again, this EEPROM and how to talk to it[br]is very well described on internet.
0:14:28.240,0:14:31.310
So I’m not gonna to describe it here.[br]If you’ve used EEPROMs
0:14:31.310,0:14:36.430
over I2C it’s likely[br]you’ve used a 24 series EEPROM.
0:14:36.430,0:14:42.790
Probably bigger ones,[br]256 bytes is pretty small.
0:14:42.790,0:14:45.930
So like a 16 bits one,
0:14:45.930,0:14:51.899
but EDID supports only[br]the 8 bits ones.
0:14:51.899,0:14:55.020
The kind of interesting part of EDID[br]is the data structure:
0:14:55.020,0:14:58.649
it’s a custom binary format that describes
0:14:58.649,0:15:02.040
what the contents of the EEPROM is.
0:15:02.040,0:15:05.410
Again, Wikipedia has a really[br]good description of this.
0:15:05.410,0:15:07.860
So I’m not gonna go[br]into much detail.
0:15:07.860,0:15:12.951
But the important thing is that it[br]describes resolution, frequency
0:15:12.951,0:15:17.589
and format for talking to the monitor.
0:15:17.589,0:15:20.720
This is really important because[br]if you try and send
0:15:20.720,0:15:25.470
the wrong resolution, frequency or format[br]the monitor is not gonna understand it.
0:15:25.470,0:15:28.579
This is kind of what EDID is used for.
0:15:28.579,0:15:32.009
sipping, sound of water bottle
0:15:32.009,0:15:38.269
So this is where things[br]start getting a bit hairy.
0:15:38.269,0:15:42.519
Presenters come up to the front and[br]the first question you see anybody ask is:
0:15:42.519,0:15:44.600
What resolution do I use?
0:15:44.600,0:15:49.919
And they get a panel like this which[br]has a bazillion of resolutions selected.
0:15:49.919,0:15:55.379
And the thing is, despite your monitor[br]saying that it supports
0:15:55.379,0:15:59.920
many formats they lie.
0:15:59.920,0:16:05.279
It turns out that projectors lie[br]a lot more than normal displays.
0:16:05.279,0:16:08.490
I don’t know why they are special.
0:16:08.490,0:16:11.560
So this is what a supported format[br]looks like.
0:16:11.560,0:16:14.769
It’s really great.
0:16:14.769,0:16:19.240
As well, I care about[br]capturing the data.
0:16:19.240,0:16:25.420
And so I want the things[br]in the format that is
0:16:25.420,0:16:27.229
easy for me to capture.
0:16:27.229,0:16:31.600
I also don’t to be scaling[br]peoples images and text
0:16:31.600,0:16:34.509
because scaling looks really bad.[br]So if someone selects
0:16:34.509,0:16:39.889
like a really low resolution and[br]we scale it up it looks really horrible.
0:16:39.889,0:16:44.300
It makes text unreadable; and[br]presenters are very denounced,
0:16:44.300,0:16:47.980
especially at technical conferences,[br]for using tiny, tiny fonts.
0:16:47.980,0:16:51.720
And so we need to use as much[br]resolution as we can.
0:16:51.720,0:16:55.649
How we solve this is we emulate[br]our own EEPROM in the FPGA
0:16:55.649,0:16:58.680
and ignore what the projector[br]tells us it can do.
0:16:58.680,0:17:03.009
We tell the presenter that this is[br]what we support.
0:17:03.009,0:17:06.689
You might notice that it kind of[br]solves the problem
0:17:06.689,0:17:10.510
of what resolution we do.
0:17:10.510,0:17:11.809
Offer a single solution…
0:17:11.809,0:17:16.499
offer a single option makes it[br]very hard to choose the wrong one.
0:17:16.499,0:17:19.839
That’s good! We solved the problem!
0:17:19.839,0:17:22.780
No, we haven’t solved the problem.
0:17:22.780,0:17:25.390
We were recording PyCon AU[br]and we found that
0:17:25.390,0:17:31.120
some Mac laptops were[br]refusing to work.
0:17:31.120,0:17:34.570
To understand the cause of this[br]you need to understand
0:17:34.570,0:17:36.970
a little bit about how the world works.
0:17:36.970,0:17:41.120
There are two major frequencies[br]in the world: 50 Hz and 60 Hz.
0:17:41.120,0:17:43.799
50 Hz is mainly used[br]in the “Rest of the World”
0:17:43.799,0:17:47.370
and 60 Hz is used in America[br]and Japan and a few other places
0:17:47.370,0:17:51.869
but that’s kind of a very rough thing.
0:17:51.869,0:17:54.559
Laptop sold in Australia,[br]Australia is 50 Hz.
0:17:54.559,0:17:57.980
It’s part of the “Rest of the World”.[br]You’d think the that the laptop
0:17:57.980,0:18:02.270
could do 50 Hz. Plus everything[br]is global these days, right?
0:18:02.270,0:18:07.950
I can plug in my power pack[br]for my laptop in the US or Australia,
0:18:07.950,0:18:11.750
like, it should work everywhere right!
0:18:11.750,0:18:15.200
No. Sad!
0:18:15.200,0:18:19.940
We solved it by claiming[br]that we were American
0:18:19.940,0:18:24.770
and supporting 60 frames per second[br]rather than 50 frames per second.
0:18:24.770,0:18:28.400
So I guess a display[br]with an American accent.
0:18:28.400,0:18:32.700
We deployed this hotfix[br]on the Friday evening.
0:18:32.700,0:18:37.279
And on Saturday all the problems[br]that we were having on Friday
0:18:37.279,0:18:41.840
went away. So this is kind of[br]the power of a open source solution
0:18:41.840,0:18:45.980
and having complete control[br]over your hardware.
0:18:45.980,0:18:49.820
Nowadays we actually offer both 60 and 50
0:18:49.820,0:18:53.600
because for display capture[br]if you’re displaying stuff
0:18:53.600,0:19:00.370
at 50 frames per second you’re[br]probably speaking a lot faster than I am.
0:19:00.370,0:19:05.650
It’s really weird, these[br]128 bytes are really hard
0:19:05.650,0:19:11.510
and the number one cause[br]of why a persons laptop
0:19:11.510,0:19:14.880
can’t talk to the projector.
0:19:14.880,0:19:18.240
It gets a trophy!
0:19:18.240,0:19:23.809
To try and figure out why that is[br]we created EDID.tv.
0:19:23.809,0:19:27.090
It’s supposed to be[br]a repository of EDID data.
0:19:27.090,0:19:30.880
There is a Summer of Code project,[br]Python/Django/Bootstrap
0:19:30.880,0:19:34.270
and an EDID grabber tool that[br]you can run on your laptop.
0:19:34.270,0:19:37.070
I’d love help making this work better.
0:19:37.070,0:19:41.700
Hasn’t had much love since[br]the Summer of Code student made that work.
0:19:41.700,0:19:45.940
But it would be really nice to have an[br]open database of everybody’s EDID data
0:19:45.940,0:19:51.250
out there. There are a bunch[br]of closed ones. I can pay to buy one,
0:19:51.250,0:19:55.630
but I’d really love to have an open one.
0:19:55.630,0:19:59.750
As well maybe we don’t need[br]the whole capture solution,
0:19:59.750,0:20:02.690
maybe you can just override the EDID.
0:20:02.690,0:20:08.700
The C3VOC here actually developed[br]a version that overrides EDID for VGA.
0:20:08.700,0:20:12.390
I have a design which works for HDMI.
0:20:12.390,0:20:20.130
It just uses a low cost microprocessor[br]to pretend to be an EEPROM.
0:20:20.130,0:20:22.890
As well, DisplayPort is not HDMI.[br]Don’t get these two confused,
0:20:22.890,0:20:25.650
they are very, very different protocols.
0:20:25.650,0:20:29.650
They have an Auxiliary Channel[br]like EDID and CEC.
0:20:29.650,0:20:34.090
I have boards to decode them[br]here at CCC.
0:20:34.090,0:20:37.140
So if you’re interested in that[br]come and talk to me
0:20:37.140,0:20:43.010
because we would really like to do[br]similar things for DisplayPort.
0:20:43.010,0:20:46.700
That is the slow speed data.
0:20:46.700,0:20:49.929
Sip from bottle
0:20:49.929,0:20:53.100
What about the high speed data?
0:20:53.100,0:20:57.500
Each pixel on your screen is
0:20:57.500,0:21:04.269
basically three colors[br]in DVI standard: Red, Green, Blue.
0:21:04.269,0:21:07.530
And each one is a byte in size.
0:21:07.530,0:21:15.540
Each of the colors is mapped to[br]a channel on the HDMI connector.
0:21:15.540,0:21:20.429
You can kind of see the Red and[br]the Green and the Blue channels.
0:21:20.429,0:21:23.130
Each channel is differential pair.
0:21:23.130,0:21:27.420
You get a Plus and a Negative[br]and a Shield.
0:21:27.420,0:21:35.460
And they use twisted pair to try[br]and reduce the noise reception of these,
0:21:35.460,0:21:38.760
because these are quite high speed.
0:21:38.760,0:21:41.860
And they have a dedicated Shield to[br]try and – again – reduce the noise
0:21:41.860,0:21:47.090
that is captured.
0:21:47.090,0:21:53.139
This is kind of where it gets to[br]the ‘differential signaling’ part
0:21:53.139,0:21:57.990
of the ‘TMDS’ that is[br]the kind of code name
0:21:57.990,0:22:05.220
for the internal protocol that is used[br]on the high speed data.
0:22:05.220,0:22:10.159
They also…[br]all these channels share a Clock.
0:22:10.159,0:22:13.830
That clock is called the Pixel Clock.
0:22:13.830,0:22:17.169
But each of these channels[br]is a serial channel.
0:22:17.169,0:22:22.929
It transmits data at 10 bits.
0:22:22.929,0:22:25.659
They… every 10 bits – sorry,
0:22:25.659,0:22:31.620
every clock cycle there are 10 bits of data[br]transmitted on each of these channels.
0:22:31.620,0:22:34.950
There is a shared clock and[br]each of the channels is running
0:22:34.950,0:22:39.340
at effectively[br]ten times that shared clock.
0:22:39.340,0:22:41.970
This is kind of what[br]the whole system looks like.
0:22:41.970,0:22:45.440
You have your Red, Green, Blue channels.
0:22:45.440,0:22:49.289
You take your 8 bits of input data[br]on each channel
0:22:49.289,0:22:53.850
and you convert it to the 10 bits
0:22:53.850,0:22:56.710
that we’re going to transmit,[br]and it goes across the cable
0:22:56.710,0:23:01.169
and then we decode it on the other side.
0:23:01.169,0:23:07.310
The question is: what does[br]the 8 bit to 10 bit encoding
0:23:07.310,0:23:11.980
look like and how do you understand that.
0:23:11.980,0:23:17.149
It is described by this diagram here.[br]It’s a bit small so I’ll bring it up.
0:23:17.149,0:23:22.749
This is what it looks like.[br]Yes… sure…
0:23:22.749,0:23:27.729
…what? This diagram – like –
0:23:27.729,0:23:32.380
I’ve spent hours looking at this,[br]and it is an extremely hard diagram
0:23:32.380,0:23:38.580
to decode.[br]It’s very, very hard to understand.
0:23:38.580,0:23:42.750
And it turns out the encoding[br]protocol is actually quite easy!
0:23:42.750,0:23:47.900
It’s three easy steps – approximately.
0:23:47.900,0:23:52.059
So I’m going to show you all how[br]to write an encoder or a decoder.
0:23:52.059,0:23:55.279
That diagram is just for the encoder.
0:23:55.279,0:24:00.729
They have a similar diagram that[br]is not the inverse of this for decoding.
0:24:00.729,0:24:03.980
Again, almost impossible to read.
0:24:03.980,0:24:07.440
The three steps: First we’re[br]going to do ‘Control’ or ‘Pixel’,
0:24:07.440,0:24:11.260
choose which one to do. Then we’re[br]going to either encode Control data
0:24:11.260,0:24:15.969
or encode Pixel data.
0:24:15.969,0:24:21.179
A couple of important points[br]to go through first:
0:24:21.179,0:24:24.480
The Input data[br]– no matter how wide it is –
0:24:24.480,0:24:29.120
is converted to 10 bit symbols.
0:24:29.120,0:24:32.059
Data goes to symbols.[br]When we’re talking about them
0:24:32.059,0:24:36.760
being transmitted we talk about them[br]in symbols, when it’s decoded into pixels
0:24:36.760,0:24:40.130
we talk about them in data.
0:24:40.130,0:24:45.760
As well, things need[br]to be kept DC-balanced.
0:24:45.760,0:24:48.480
I’ve rushed ahead.
0:24:48.480,0:24:52.850
The question is: “Why 10 bits?”[br]Our pixels were 8 bits.
0:24:52.850,0:24:56.070
I will explain why[br]in the Pixel Data section.
0:24:56.070,0:24:59.060
But it’s important that all our symbols[br]are the same size.
0:24:59.060,0:25:05.159
We’re always transmitting 10 bits[br]every clock cycle.
0:25:05.159,0:25:08.530
Keeping DC-balanced:
0:25:08.530,0:25:12.879
long runs of 1s and 0s are bad.
0:25:12.879,0:25:15.909
There are lots of reasons for this.
0:25:15.909,0:25:21.809
I tend to think of it like[br]HDMI isn’t AC coupled
0:25:21.809,0:25:27.280
but you can kind of think of it[br]like AC coupled.
0:25:27.280,0:25:30.389
It’s not to recover Clock.
0:25:30.389,0:25:35.419
We have a clock pair that is used[br]to give our Clock signal.
0:25:35.419,0:25:38.059
There are lots of lies on internet[br]that say that the reason
0:25:38.059,0:25:42.649
we’re going to keep DC balance[br]is because of Clock.
0:25:42.649,0:25:47.549
But no, that’s not the case.
0:25:47.549,0:25:52.320
So what does DC balance mean?
0:25:52.320,0:25:57.110
A symbol which has lots of 1s[br]or lots of 0s
0:25:57.110,0:26:00.870
is going to be considered DC-biased
0:26:00.870,0:26:05.270
if it has more 1s than 0s.
0:26:05.270,0:26:08.769
This is kind of what it’s like:[br]this symbol here
0:26:08.769,0:26:12.530
has lots of 1s and if you add up[br]all the 1s you can see it’s got
0:26:12.530,0:26:16.919
quite a positive bias.[br]If it was inverse and had lots of 0s
0:26:16.919,0:26:19.509
it would have a negative DC bias.
0:26:19.509,0:26:26.610
That cause… that DC bias over time[br]causes us problems.
0:26:26.610,0:26:32.380
That are the two important things we have[br]to keep in mind when looking at the rest.
0:26:32.380,0:26:34.429
sound of bottle sip
0:26:34.429,0:26:37.710
The first thing we need to figure out is[br]are we transmitting Control data
0:26:37.710,0:26:39.940
or Pixel data.
0:26:39.940,0:26:44.010
Turns out that what is happening[br]in your display is,
0:26:44.010,0:26:47.089
we are transmitting something[br]that’s actually bigger
0:26:47.089,0:26:50.909
than what you[br]see on your screen.
0:26:50.909,0:26:57.370
This not the scale. The Control data[br]periods are much, much smaller.
0:26:57.370,0:27:06.230
The Control data is in orange[br]and the Pixel data is in purple-pink.
0:27:06.230,0:27:12.260
So why does this exist?[br]It exists because of old CRT monitors.
0:27:12.260,0:27:16.620
And for those in the audience[br]who where kind of born after CRT monitors,
0:27:16.620,0:27:19.720
this is what they look like.
0:27:19.720,0:27:23.350
The way they work is,[br]they have an electron beam
0:27:23.350,0:27:27.919
that scans across,[br]highlighting the phosphorus.
0:27:27.919,0:27:34.620
This electron beam can’t just be…[br]get back to other side of the screen
0:27:34.620,0:27:39.049
straight away, or get back to the top of[br]the screen. And so these periods
0:27:39.049,0:27:43.639
where we are transmitting Control data[br]was to allow the electron beam
0:27:43.639,0:27:47.470
to get back to the location[br]where it needed to start
0:27:47.470,0:27:53.160
transmitting the next set of data.
0:27:53.160,0:27:56.079
That’s why it exists.[br]Why do we care?
0:27:56.079,0:27:59.090
Because the encoding schemes[br]for Control and Pixel data
0:27:59.090,0:28:03.820
are actually quite different.
0:28:03.820,0:28:07.150
This is the main difference.[br]I’m going to come back to this slide
0:28:07.150,0:28:11.509
a bit later. But again, the[br]important thing to see here is
0:28:11.509,0:28:15.300
that despite the encoding scheme[br]being quite different
0:28:15.300,0:28:21.630
the output is 10 bits in size.
0:28:21.630,0:28:25.389
That first step – choosing whether[br]it’s Pixel or Control data –
0:28:25.389,0:28:30.110
is described by this bit of the diagram.[br]You might notice that’s
0:28:30.110,0:28:34.309
not the first thing in the diagram.
0:28:34.309,0:28:37.900
How do you convert Control data[br]to Control symbols?
0:28:37.900,0:28:41.419
First we need to know what[br]Control data is. There are two bits,
0:28:41.419,0:28:44.500
there is the HSync and the VSync signal.
0:28:44.500,0:28:51.269
They provide basically[br]the horizontal and vertical pixel sizes.
0:28:51.269,0:28:55.460
They are kind of left over from VGA.[br]We don’t actually need them
0:28:55.460,0:29:01.100
in HDMI or DVI to know[br]where the edges are
0:29:01.100,0:29:06.710
because we can tell the difference[br]between Control and Pixel data.
0:29:06.710,0:29:11.890
But they kind of still exist[br]because of backwards compatibility.
0:29:11.890,0:29:15.860
This means that we have two bits of data[br]that we need to convert to 10 bits of data.
0:29:15.860,0:29:22.409
So, a 2b/10b scheme.
0:29:22.409,0:29:27.270
How they do it is they just hand-picked[br]four symbols that were going to be
0:29:27.270,0:29:30.380
these Control data symbols.
0:29:30.380,0:29:35.020
These are the four symbols. There’s[br]some interesting properties with them.
0:29:35.020,0:29:38.720
They are chosen to be DC-balanced.[br]They roughly have the same number
0:29:38.720,0:29:46.870
of 0s and 1s. So we don’t have to worry about[br]the DC bias with these symbols very much.
0:29:46.870,0:29:51.740
They are also chosen to have[br]seven or more transitions from 0 to 1
0:29:51.740,0:29:58.610
in them. This number of transitions
0:29:58.610,0:30:03.230
is used to understand[br]the phase relationship
0:30:03.230,0:30:07.020
of the different channels.[br]So if you remember this diagram,
0:30:07.020,0:30:12.950
we have a cable going between[br]the transmitter and the transceiver.
0:30:12.950,0:30:16.360
These are, again, very high speed signals.
0:30:16.360,0:30:22.130
And even if the transmitter was[br]transmitting everything at the same time,
0:30:22.130,0:30:28.240
the cable isn’t ideal and might[br]delay some of the symbols.
0:30:28.240,0:30:32.880
The bits on one channel[br][might take] longer than others.
0:30:32.880,0:30:37.320
By having lots of these transmissions[br]we can actually find
0:30:37.320,0:30:41.590
the phase relationship between[br]each of the channels and then
0:30:41.590,0:30:47.559
recover the data. And so[br]that’s why these Control symbols
0:30:47.559,0:30:52.400
have a large number[br]of transitions in them.
0:30:52.400,0:30:56.750
More on that later when we get to the[br]implementation. And I’m running out’ time.
0:30:56.750,0:31:01.159
This part of the diagram is the[br]Control data encoding.
0:31:01.159,0:31:04.190
sip from bottle
0:31:04.190,0:31:06.990
What about Pixel data[br]and the Pixel symbols?
0:31:06.990,0:31:13.740
Again, in DVI each channel[br]of the Pixel is 8 bits.
0:31:13.740,0:31:17.199
And the encoding scheme is described[br]by the rest of the diagram.
0:31:17.199,0:31:22.480
But again, it’s actually[br]really, really simple.
0:31:22.480,0:31:26.520
This encoding scheme is called 8b/10b,
0:31:26.520,0:31:30.030
because it takes 8 bits[br]converting it to 10 bits.
0:31:30.030,0:31:33.880
However, there is a huge danger[br]here because IBM also invented
0:31:33.880,0:31:37.480
the 8b/10b scheme[br]that is used in everything.
0:31:37.480,0:31:41.080
This is used in DisplayPort, it’s used[br]in PCI Express, it’s used in SATA,
0:31:41.080,0:31:43.929
it’s used in pretty much everything[br]on the planet.
0:31:43.929,0:31:48.259
This is not the encoding TDMS uses.
0:31:48.259,0:31:52.490
You can lose a lot of time[br]trying to map this diagram
0:31:52.490,0:31:56.560
to the IBM coding scheme,[br]and going these are not the same.
0:31:56.560,0:32:03.490
That is because they’re not the same.[br]This is a totally different coding scheme.
0:32:03.490,0:32:07.570
Encoding Pixel data is a two-step process.[br]I did say it was three-ish steps
0:32:07.570,0:32:12.270
to do this.[br]The first step is we want to reduce
0:32:12.270,0:32:18.110
the transitions in the data.
0:32:18.110,0:32:20.429
How do we do this? –[br]Sorry, why do we do this?
0:32:20.429,0:32:23.570
Because this, again, is[br]a high speed channel.
0:32:23.570,0:32:27.529
We want to reduce the cross-talk[br]between the lanes.
0:32:27.529,0:32:30.669
They are actually quite close[br]to each other.
0:32:30.669,0:32:34.779
So by reducing the number[br]of transitions we can reduce
0:32:34.779,0:32:40.130
the probability that the signal propagates
0:32:40.130,0:32:45.760
from one channel to the next.[br]And how we do it?
0:32:45.760,0:32:49.789
We’re gonna choose one[br]of two encoding schemes.
0:32:49.789,0:32:54.000
An XOR encoding scheme[br]or an XNOR encoding scheme.
0:32:54.000,0:32:57.740
How do we do the XOR encoding scheme?[br]It’s actually pretty simple.
0:32:57.740,0:33:00.679
We set the Encoded Bit[br]same as the first Data Bit
0:33:00.679,0:33:04.310
and then the next Encoded Bit[br]is the first Encoded Bit
0:33:04.310,0:33:09.649
XORed with the Data bit.
0:33:09.649,0:33:14.000
And then we just repeat until[br]we’ve done the 8 bits.
0:33:14.000,0:33:16.159
So this is how we do the XOR encoding.
0:33:16.159,0:33:20.149
The XNOR encoding is the same process,[br]except instead of using XOR
0:33:20.149,0:33:24.500
it uses XNOR.
0:33:24.500,0:33:29.210
How do we choose[br]which one of these to use?
0:33:29.210,0:33:35.019
If the Input Data byte[br]has fewer than four 1s
0:33:35.019,0:33:39.919
we use the XOR. If it has more[br]than four 1s we use the XNOR.
0:33:39.919,0:33:43.059
And then there’s a tie-break (?)[br]if you have even.
0:33:43.059,0:33:47.850
The important thing here is that this[br]method is determined by the Data byte only.
0:33:47.850,0:33:53.000
There is no hidden state here[br]or continuous change.
0:33:53.000,0:33:59.710
Every pixel has a one-to-one[br]mapping to an encoding.
0:33:59.710,0:34:04.260
Then we append a bit[br]on the end that indicates
0:34:04.260,0:34:09.030
whether we chose XOR,[br]XNOR encoding of that data.
0:34:09.030,0:34:15.219
And so that converts[br]our 8 bits Input Pixels
0:34:15.219,0:34:21.500
to 9 bits of encoded data, effectively[br]our 8-bit encoded sequence
0:34:21.500,0:34:27.720
and then one bit to indicate whether[br]we chose XOR, or XNOR encoding
0:34:27.720,0:34:34.239
for that Data bit. So that’s it there.
0:34:34.239,0:34:38.060
This encoding is actually very good[br]at reducing transitions.
0:34:38.060,0:34:43.520
On average, we had roughly[br]eight transitions previously,
0:34:43.520,0:34:48.900
now we have roughly three-ish,[br]so it’s pretty cool.
0:34:48.900,0:34:51.220
I have no idea how they figured this out.
0:34:51.220,0:34:55.570
I’m assuming some very smart[br]mathematicians where involved
0:34:55.570,0:35:00.330
because discovering this is beyond me.
0:35:00.330,0:35:02.280
And that describes the top part[br]of this process.
0:35:02.280,0:35:04.410
sounds of scratching nose and beard
0:35:04.410,0:35:11.660
This is where, in the TMDS, the[br]Transition Minimization comes from,
0:35:11.660,0:35:14.220
that step there in the encoding process.
0:35:14.220,0:35:16.310
But there is still one more step.
0:35:16.310,0:35:22.370
We need to keep the channel[br]DC-balanced, as I explained earlier.
0:35:22.370,0:35:27.860
How can we do that? Because[br]not all pixels are guaranteed to be
0:35:27.860,0:35:32.160
at zero DC bias[br]like the Control symbols are.
0:35:32.160,0:35:37.320
We do it by keeping a running count[br]of the DC bias we have,
0:35:37.320,0:35:41.500
and then, if we have a positive DC bias
0:35:41.500,0:35:45.890
and the symbol is also[br]positively biased, we invert it.
0:35:45.890,0:35:51.770
Or, if we have a negative DC bias[br]and the symbol has a negative DC bias,
0:35:51.770,0:35:55.910
we invert it.[br]And the reason we do this is
0:35:55.910,0:36:00.830
because when we invert a symbol we[br]convert all the 1s to 0s which means
0:36:00.830,0:36:05.790
a negative DC bias[br]becomes a positive DC bias.
0:36:05.790,0:36:10.800
As I said, we chose – because we are already[br]negative and the thing was negative –
0:36:10.800,0:36:17.940
we convert it to plus. It means we’re[br]going to drive the running DC bias value
0:36:17.940,0:36:22.620
back towards zero.[br]We might overshoot, but the next stage
0:36:22.620,0:36:27.100
we’ll keep trying to oscillate up and[br]down, and on average over time
0:36:27.100,0:36:31.380
we keep a DC bias of zero.
0:36:31.380,0:36:36.870
And as I said. Then, to indicate[br]whether or not we inverted
0:36:36.870,0:36:42.740
or kept… the…[br]straight through we inverted,
0:36:42.740,0:36:48.080
we add another bit on the end.[br]So that’s how get our 10 bit
0:36:48.080,0:36:53.780
encoding scheme.[br]We have the 8 bits of encoded data,
0:36:53.780,0:36:58.800
then one bit indicating whether or not[br]it used XOR/XNOR encoding,
0:36:58.800,0:37:04.030
and then one bit to indicate whether[br]or not we inverted the symbol.
0:37:04.030,0:37:09.970
That describes this bottom part[br]of the chart.
0:37:09.970,0:37:14.510
Now you can see partly[br]why this chart is kind of confusing.
0:37:14.510,0:37:18.840
It’s no way in what I think[br]of a what’s a logical diagram.
0:37:18.840,0:37:21.610
This might be how you implement it[br]in hardware if you really understand
0:37:21.610,0:37:28.990
the protocol, but not a very good diagram[br]for explaining what’s going on. And…
0:37:28.990,0:37:31.590
sip from bottle
0:37:31.590,0:37:34.190
As you see it’s actually pretty simple.
0:37:34.190,0:37:40.360
In summary this is[br]the interesting information
0:37:40.360,0:37:45.390
about the two different encoding schemes.
0:37:45.390,0:37:49.350
Because we minimized[br]the transitions in the Pixel data
0:37:49.350,0:37:53.020
we can actually tell[br]Control and Pixel data apart
0:37:53.020,0:37:56.360
by looking at how many transitions[br]are in the symbol.
0:37:56.360,0:38:00.840
If it has six or more transitions[br]it must be a Control symbol.
0:38:00.840,0:38:06.040
If it has four or less[br]it must be a Pixel symbol.
0:38:06.040,0:38:09.610
You now know[br]how to encode TDMS data
0:38:09.610,0:38:12.290
and how to decode TDMS data
0:38:12.290,0:38:18.070
because if you want to decode[br]you just do the process backwards.
0:38:18.070,0:38:25.480
Congratulations![br]How do you actually implement this?
0:38:25.480,0:38:28.290
You can just write the XOR logic
0:38:28.290,0:38:31.252
and a little counter[br]that keeps track of the DC bias
0:38:31.252,0:38:34.780
and all that type of thing[br]in FPGA.
0:38:34.780,0:38:38.690
I’m not going to describe that[br]because I don’t have much time.
0:38:38.690,0:38:43.130
But if you followed the process[br]that I have given you
0:38:43.130,0:38:45.980
it should be pretty easy.
0:38:45.980,0:38:50.990
But… and this is what we use currently.
0:38:50.990,0:38:53.700
You could actually use a lookup table.[br]What we are doing is
0:38:53.700,0:38:58.380
converting 8 bits of data[br]to 10 bits of data.
0:38:58.380,0:39:03.760
That is a lookup table process,[br]pretty easy.
0:39:03.760,0:39:08.900
FPGAs are really good at[br]doing ‘lookup table’-type processes,
0:39:08.900,0:39:13.010
and it also allows you then[br]to extend this system
0:39:13.010,0:39:18.150
to those other protocols[br]like the 4b/10b that is used
0:39:18.150,0:39:20.890
for the Auxiliary data.
0:39:20.890,0:39:24.770
So we are looking at that in the future.[br]It uses a few more resources
0:39:24.770,0:39:28.020
but it’s a lot more powerful.
0:39:28.020,0:39:32.790
This is kind of what your encoder[br]will look like, and your decoder.
0:39:32.790,0:39:36.650
It’s quite simple,[br]it takes in your 10 bits of data
0:39:36.650,0:39:39.400
and outputs either[br]your 8 bits of Pixel data
0:39:39.400,0:39:43.700
or your 2 bits of Control data[br]and the data type.
0:39:43.700,0:39:47.190
This is kind of what if you went[br]into our design and looked at it
0:39:47.190,0:39:49.570
at high level, in the schematic,
0:39:49.570,0:39:52.220
you’d probably see a block[br]that looks like this.
0:39:52.220,0:39:56.430
The encoder is slightly more complicated[br]because you also have the DC bias count
0:39:56.430,0:40:00.990
that you have to keep track of.[br]But, again,
0:40:00.990,0:40:03.720
the data goes in[br]and the data comes out.
0:40:03.720,0:40:08.810
That’s simple, right?
0:40:08.810,0:40:12.260
This kind of extends to Auxiliary data,[br]or if you get an error,
0:40:12.260,0:40:15.490
like if you…[br]There are 124 symbols
0:40:15.490,0:40:18.930
that you can have in 10 bits of data.
0:40:18.930,0:40:22.160
Not all of them are valid.[br]So if you get one of these invalid symbols
0:40:22.160,0:40:25.770
you know you have an error.
0:40:25.770,0:40:30.270
However, things happen quite quickly
0:40:30.270,0:40:34.520
when you times them by ten.[br]And so our Pixel Clock
0:40:34.520,0:40:39.150
for 640x480 is 25 MHz.[br]When you times that by ten
0:40:39.150,0:40:45.000
you get 250 MBits per channel.[br]When you’re doing 720p
0:40:45.000,0:40:48.580
you’re doing 750 MBits per channel.
0:40:48.580,0:40:53.810
And 1080p is at 1500 MBits per channel.
0:40:53.810,0:40:59.080
An FPGA is fast, but[br]they’re not really that fast
0:40:59.080,0:41:03.950
at a range that I can afford to buy.[br]I’m sure the military has ones
0:41:03.950,0:41:08.480
that go this fast, but[br]I’m not as rich as them.
0:41:08.480,0:41:12.180
But they do include a nice hack[br]to solve this.
0:41:12.180,0:41:15.310
They are called SerDes.[br]They basically turn parallel data
0:41:15.310,0:41:18.790
into serial data.
0:41:18.790,0:41:21.830
This is what the boxes look like.
0:41:21.830,0:41:24.230
You give them your TDMS parallel data
0:41:24.230,0:41:27.940
and they convert it to[br]high speed serial data for you.
0:41:27.940,0:41:32.540
They are a little bit fiddly to use[br]and your best option is to go and find
0:41:32.540,0:41:37.400
a person who has already configured this[br]for your FPGA
0:41:37.400,0:41:39.900
and follow what they do.
0:41:39.900,0:41:44.430
“Hamster” – Mike “Hamster” Field – has[br]a really good documentation on
0:41:44.430,0:41:49.940
how to use these in a Spartan6.[br]These are also unique to your FPGA,
0:41:49.940,0:41:54.140
so different FPGAs are going to have[br]different control schemes.
0:41:54.140,0:41:56.900
But if you are using a Spartan6
0:41:56.900,0:42:02.390
then go and look up what[br]Mike “Hamster” Field is
0:42:02.390,0:42:07.770
doing for configuring these.
0:42:07.770,0:42:14.190
Remember how I said,[br]our system has a serial console.
0:42:14.190,0:42:18.630
Because we have that system[br]we can actually delve quite deep
0:42:18.630,0:42:22.910
into what’s happening[br]internally in the system.
0:42:22.910,0:42:25.110
sip from bottle
0:42:25.110,0:42:32.920
And print it out.[br]This is debugging from one of our systems.
0:42:32.920,0:42:35.140
You can see…
0:42:35.140,0:42:40.550
The first thing is the phase relationship[br]between each of the channels.
0:42:40.550,0:42:44.790
The next one is whether[br]we’re getting valid data
0:42:44.790,0:42:49.770
on each of the channels and then[br]we’ve got the error rate for that channel,
0:42:49.770,0:42:54.370
whether all channels synchronized,[br]and then some resolution information.
0:42:54.370,0:43:00.200
You can see that this has got[br]a 74 MHz Pixel Clock.
0:43:00.200,0:43:05.080
There are three columns because[br]there is Red, Green and Blue channels.
0:43:05.080,0:43:09.230
This give us some very interesting[br]debugging capabilities.
0:43:09.230,0:43:13.210
If you plug in a cable[br]and you’re getting errors
0:43:13.210,0:43:17.240
on the Blue channel and nowhere else
0:43:17.240,0:43:21.560
it’s highly likely there’s[br]something wrong with that cable.
0:43:21.560,0:43:25.500
This is a very powerful tool[br]that allows us to figure out
0:43:25.500,0:43:29.740
what’s going wrong in a system.
0:43:29.740,0:43:35.270
It’s something you can’t really get[br]with the commercial versions of this.
0:43:35.270,0:43:39.060
But what about errors?[br]Everything I’m talking about now
0:43:39.060,0:43:42.040
is a little bit experimental,[br]we haven’t actually implemented this.
0:43:42.040,0:43:45.700
But it’s some ideas about[br]what we can do because we now
0:43:45.700,0:43:49.590
have complete control of our decoder.
0:43:49.590,0:43:54.220
As I said, there’s 124 possible choices[br]for 10 bit symbols,
0:43:54.220,0:43:58.220
of which 460 are valid Pixel symbols,
0:43:58.220,0:44:02.390
4 are valid Control symbols[br]and 560 symbols
0:44:02.390,0:44:05.060
should never ever be seen no matter what.
0:44:05.060,0:44:11.890
That’s like 56% of our space[br]that should never be seen.
0:44:11.890,0:44:16.330
But it’s actually better than that![br]We know because of the running DC bias
0:44:16.330,0:44:25.050
that there are 256 valid Pixel symbols
0:44:25.050,0:44:31.150
at any one point. You can’t have the…[br]if you’ve got a negative DC bias
0:44:31.150,0:44:37.240
you can’t have a Pixel symbol[br]which continues to drive you negative.
0:44:37.240,0:44:43.710
Actually, 74% of our space at any time
0:44:43.710,0:44:48.030
is not allowed to exist.
0:44:48.030,0:44:52.070
This means that a huge number[br]of the invalid symbols
0:44:52.070,0:44:56.180
are only near one other valid symbol.
0:44:56.180,0:45:01.530
And so we can actually correct them![br]We can go: “This symbol must have been
0:45:01.530,0:45:04.950
this other symbol,[br]because it’s not a valid symbol,
0:45:04.950,0:45:08.840
it must be a bit error[br]from this other symbol.”
0:45:08.840,0:45:12.870
So we can correct these errors.[br]This is quite cool.
0:45:12.870,0:45:19.030
We can correct about 70% of
0:45:19.030,0:45:21.970
single bit flip errors in Pixel data.
0:45:21.970,0:45:28.510
But sadly there is some that we can’t.
0:45:28.510,0:45:34.960
But we can detect that we got[br]a invalid Pixel data.
0:45:34.960,0:45:40.320
So the fact that there is an error[br]is important.
0:45:40.320,0:45:43.910
In this case we’ve got two pixels[br]that we received correctly
0:45:43.910,0:45:49.030
and we got a pixel that we know[br]is a invalid value
0:45:49.030,0:45:53.550
and then two more pixels[br]that we received correctly.
0:45:53.550,0:45:55.180
You can imagine this is a Blue channel,
0:45:55.180,0:45:59.040
so the first ones were not very blue.
0:45:59.040,0:46:03.010
Then there’s the decoded value for this is
0:46:03.010,0:46:07.290
very, very blue, like very light blue[br]and then some other ones.
0:46:07.290,0:46:09.950
This looks really bad, right?
0:46:09.950,0:46:15.480
This was probably a whole blue block.
0:46:15.480,0:46:20.180
One pixel difference[br]of that big, that size,
0:46:20.180,0:46:24.030
is probably not a valid value,
0:46:24.030,0:46:26.440
and so we can cover them up!
0:46:26.440,0:46:29.680
We can go…[br]the two pixels on either side
0:46:29.680,0:46:32.320
and average them and fix that pixel.
0:46:32.320,0:46:38.250
This allow us to correct a whole bunch[br]more of errors that are occurring.
0:46:38.250,0:46:41.460
And as we’re about to take this data
0:46:41.460,0:46:45.940
and run it through a JPEG encoder
0:46:45.940,0:46:49.620
this doesn’t actually affect[br]the quality of the output
0:46:49.620,0:46:53.150
all that much and allows to fix[br]things that would otherwise
0:46:53.150,0:46:59.810
be giant glaring glitches in the output.
0:46:59.810,0:47:02.440
That’s some interesting information about
0:47:02.440,0:47:09.490
how you do TDMS decoding[br]and how we can fix some errors.
0:47:09.490,0:47:12.560
The thing is, we can do it[br]even better than this
0:47:12.560,0:47:15.530
because it’s an open source project.
0:47:15.530,0:47:19.600
Maybe you have some idea[br]about how we can improve
0:47:19.600,0:47:22.890
the SerDes performance.[br]Maybe you have an idea about
0:47:22.890,0:47:28.870
how to do TDMS decoding on[br]much lower power devices
0:47:28.870,0:47:33.540
than we use. It’s open source![br]You can look at the code
0:47:33.540,0:47:39.100
and you can improve it.[br]And we would love you to do it!
0:47:39.100,0:47:43.450
The thing is that I have a lot of hardware[br]but not much time.
0:47:43.450,0:47:45.710
If you have lots of time[br]and not much hardware,
0:47:45.710,0:47:49.940
I think I can solve this problem.
0:47:49.940,0:47:54.790
These are links to the HDMI2USB project
0:47:54.790,0:47:58.840
and the TimVideos project;[br]and all our code, our hardware,
0:47:58.840,0:48:04.620
everything is on GitHub[br]under open source licenses.
0:48:04.620,0:48:11.110
And here is some bonus screen shots that[br]I wasn’t able to fit in other locations.
0:48:11.110,0:48:13.920
You can see these small errors.
0:48:13.920,0:48:16.840
That one was kind of a big error.
0:48:16.840,0:48:25.880
This is what happens when[br]your DDR memory is slightly broken.
0:48:25.880,0:48:31.800
Yeah…[br]but – yeah!
0:48:31.800,0:48:35.030
And that is my talk!
0:48:35.030,0:48:42.990
applause
0:48:42.990,0:48:45.250
Herald: Excellent![br]Thank you very much, mithro.
0:48:45.250,0:48:48.850
As you’ve noticed, we have a couple of[br]microphones standing around in the room.
0:48:48.850,0:48:52.960
If you have any questions for mithro[br]please line up behind the microphones
0:48:52.960,0:48:57.950
and I will allow you to ask the questions.[br]We have question from the Internet!?
0:48:57.950,0:49:01.700
Signal Angel: Yes, thank you![br]Do you know if normal monitors
0:49:01.700,0:49:04.650
do similar error recovery or hiding?
0:49:04.650,0:49:08.910
Tim: I know of no commercial[br]implementation that does
0:49:08.910,0:49:12.720
any type of error correction.[br]The solution for the commercial guys
0:49:12.720,0:49:19.840
is to effectively never get errors.
0:49:19.840,0:49:24.170
They can do that because
0:49:24.170,0:49:26.700
they don’t have to deal with[br]the angry speakers on the ground
0:49:26.700,0:49:30.970
going wild as my slides look weird.
0:49:30.970,0:49:34.760
And, as well, they are probably working[br]with better quality hardware
0:49:34.760,0:49:39.120
than we are using. We’re trying[br]to make things as cheap as possible.
0:49:39.120,0:49:43.720
And so we are pushing the boundaries[br]of a lot of the devices we are using.
0:49:43.720,0:49:48.250
So we are more likely to get[br]errors than they are.
0:49:48.250,0:49:51.580
Herald: We have quite a lot of questions.[br]Remember – questions, not comments!
0:49:51.580,0:49:55.520
Microphone number 1, please!
0:49:55.520,0:50:11.260
rustling sound from audience[br]coughing
0:50:11.260,0:50:12.830
Tim: Yes!
0:50:12.830,0:50:18.180
unrecognizable question from audience
0:50:18.180,0:50:20.920
Sorry, I don’t quite understand[br]what’s going on! chuckles
0:50:20.920,0:50:27.400
Herald: Do we have a translation?
0:50:27.400,0:50:29.890
Voice from audience: Audio Angel!
0:50:29.890,0:50:34.220
Tim: Audio problem?
0:50:34.220,0:50:45.490
Herald speaks to person[br]in front of stage in German
0:50:45.490,0:50:51.530
Tim: I’ll be around afterwards,[br]If you want to chat to me, ahm…
0:50:51.530,0:50:57.110
Herald: And we might do that… ah…[br]write you on the computer afterwards.
0:50:57.110,0:51:01.530
Second question from[br]microphone number 3, please!
0:51:01.530,0:51:05.500
Question: Hello? Ah, yes. Can you[br]determine the quality of a HDMI cable,
0:51:05.500,0:51:08.520
e.g. by measuring bit error rate[br]of each three pairs
0:51:08.520,0:51:11.350
and also some jitter on the clock,[br]and that kind of…?
0:51:11.350,0:51:13.970
Tim: Yes we can!
0:51:13.970,0:51:18.100
The quality of a HDMI cable should be[br]they’re zero bit errors.
0:51:18.100,0:51:23.540
So anything that has non-zero bit errors[br]we chop up and throw away.
0:51:23.540,0:51:27.870
This gets interesting[br]when you have very long cables.
0:51:27.870,0:51:31.150
We can actually see that[br]the longer the cable is
0:51:31.150,0:51:37.160
the harder for them[br]to keep zero bit errors.
0:51:37.160,0:51:43.230
So yes, we can kind of judge[br]the quality of the cable.
0:51:43.230,0:51:47.500
But it’s also hard because
0:51:47.500,0:51:51.460
it depends on what the sender is doing.
0:51:51.460,0:51:54.750
If the sender is of a lower quality
0:51:54.750,0:51:58.070
and the cable is low quality[br]you might get bit errors.
0:51:58.070,0:52:00.400
But if the sender is of a high quality
0:52:00.400,0:52:06.810
and the cable is of a lower quality
0:52:06.810,0:52:11.330
they might cancel each other out[br]and still be fine.
0:52:11.330,0:52:16.330
We can’t just go: “This is a good cable”
0:52:16.330,0:52:20.880
because we don’t actually have[br]any control over our…
0:52:20.880,0:52:25.810
how powerful our sender is on this device.
0:52:25.810,0:52:28.040
If we could kind of turn down the sender
0:52:28.040,0:52:31.210
and see where things start going wrong[br]that would be pretty cool.
0:52:31.210,0:52:34.760
If anybody wants to[br]look at building such a device
0:52:34.760,0:52:37.910
I’d love to help you do that.
0:52:37.910,0:52:41.300
Herald: We have another question[br]from microphone number 5.
0:52:41.300,0:52:44.880
Question: Your hardware,[br]the HDMI2USB hardware…
0:52:44.880,0:52:47.900
Tim: Yes![br]Question: Is it available for simply ordering
0:52:47.900,0:52:51.970
or has it to be solder soldered by hand,[br]or…
0:52:51.970,0:52:56.280
Tim: You can not solder this board by[br]hand unless you are much, much better
0:52:56.280,0:53:01.790
than I am. It uses Ball Grid Array parts[br]because it’s an FPGA.
0:53:01.790,0:53:04.530
This is one here.[br]You can buy them.
0:53:04.530,0:53:10.020
We’re working with a manufacturer in India[br]who builds them for us.
0:53:10.020,0:53:15.280
We work with them,[br]and it was pretty awesome.
0:53:15.280,0:53:17.540
We’re also working on new hardware.[br]I’ve got a whole bunch
0:53:17.540,0:53:21.860
of FPGA hardware here[br]that you can come and have a look at
0:53:21.860,0:53:25.920
and I’ll probably move it out[br]into the hallway afterwards.
0:53:25.920,0:53:30.490
Again, if you’re interested[br]in the hardware and you have a use case,
0:53:30.490,0:53:35.100
chat to me![br]Because I like to solve problems
0:53:35.100,0:53:39.460
of people who are not having hardware[br]and my employer pays me too much.
0:53:39.460,0:53:43.210
So I get to use my discretion refunds
0:53:43.210,0:53:47.710
for helping out people[br]doing open source stuff.
0:53:47.710,0:53:54.500
applause
0:53:54.500,0:53:59.220
Herald: We have at least four more[br]questions. Microphone number 2, please!
0:53:59.220,0:54:05.030
Question: Do you think it would be[br]possible to get an 1080p image
0:54:05.030,0:54:09.780
out of the open source[br]hardware board you use?
0:54:09.780,0:54:16.960
Tim: Yes, I do, but it requires[br]us to do some hard work
0:54:16.960,0:54:19.070
that we haven’t had time to do yet.
0:54:19.070,0:54:26.990
And for us 720p at 60 frames[br]per second is good enough.
0:54:26.990,0:54:32.340
The USB connection
0:54:32.340,0:54:36.290
is limited in bandwidth because[br]we don’t have an H.264 encoder,
0:54:36.290,0:54:43.020
we only have MJPEG. If somebody wants[br]to write us a open source, say, WebM
0:54:43.020,0:54:47.000
rather than H.264 encoder
0:54:47.000,0:54:50.710
that might start become more interesting.[br]We also have ethernet, Gigabit ethernet,
0:54:50.710,0:54:56.470
on this board. It should be pretty ease[br]to stream the data out the ethernet.
0:54:56.470,0:54:59.140
I, again, need help.
0:54:59.140,0:55:01.700
The ethernet controller works.[br]We can telnet into the board
0:55:01.700,0:55:07.050
and control it via Telnet.[br]We just need somebody to
0:55:07.050,0:55:11.470
actually connect the data,[br]the high speed data side up.
0:55:11.470,0:55:14.780
We use it for debugging and stuff.
0:55:14.780,0:55:18.720
Mike “Hamster” Field again,[br]really big thank you to him,
0:55:18.720,0:55:22.490
he is an amazing designer,
0:55:22.490,0:55:28.780
he built 1080p60 that is[br]a little bit out-of-spec
0:55:28.780,0:55:33.120
but actually works really well on hardware
0:55:33.120,0:55:38.870
that is almost identical to ours.[br]He also did the DisplayPort,
0:55:38.870,0:55:43.490
like a 4k-DisplayPort which[br]we can do on our board.
0:55:43.490,0:55:49.630
If you only need one or two[br]of the 1080p things
0:55:49.630,0:55:52.670
DisplayPort connectors can be[br]converted to HDMI quite easily
0:55:52.670,0:55:56.500
and you can do that on them.
0:55:56.500,0:55:58.850
Yes, I think that’s possible,[br]but again:
0:55:58.850,0:56:05.900
open source … hobbyist …[br]need developers …
0:56:05.900,0:56:07.990
Herald: We’ll take one question[br]from the internet!
0:56:07.990,0:56:12.870
Signal Angel: Thank you. Have you[br]considered JPEG2000?
0:56:12.870,0:56:17.860
Tim: No, I’ve not. And the main reason[br]is that I want to be a webcam.
0:56:17.860,0:56:25.780
I want to pretend to be a webcam. The UVC[br]standard, which is the USB webcam standard,
0:56:25.780,0:56:30.720
does not support JPEG2000.
0:56:30.720,0:56:34.260
There’s no reason we couldn’t support[br]JPEG2000 when connected to Linux,
0:56:34.260,0:56:38.770
like we could fix the Linux driver[br]to add JPEG2000 support.
0:56:38.770,0:56:44.890
Again, I don’t know if there’s any good[br]open source FPGA implementations
0:56:44.890,0:56:52.670
of JPEG2000? So, that’s also a blocker.
0:56:52.670,0:56:57.500
But if you’re interested in helping out[br]– come and talk to me.
0:56:57.500,0:57:01.780
As I said, I would very much love
0:57:01.780,0:57:06.600
to chat to you and solve[br]the problems you’re having
0:57:06.600,0:57:11.540
with getting-going.[br]As well, we have t-shirts.
0:57:11.540,0:57:16.410
I’m wearing a t-shirt that we have, and[br]I will send everybody who contributes
0:57:16.410,0:57:23.780
a t-shirt. Whether that’s fixing our[br]website, helping on documentation,
0:57:23.780,0:57:28.050
helping people on IRC[br]getting setup, anything.
0:57:28.050,0:57:34.320
You don’t need to be an expert[br]on FPGA stuff to help out.
0:57:34.320,0:57:38.410
And we also are working on[br]a little project to run MicroPython
0:57:38.410,0:57:43.160
on FPGAs. So if you’re really into Python[br]and you like MicroPython
0:57:43.160,0:57:48.060
I would love to help you help us do that.
0:57:48.060,0:57:53.260
It’s kind of working. We just need more[br]powerful (?) support. So.
0:57:53.260,0:57:56.230
Herald: We have two more questions from[br]microphone number 1.
0:57:56.230,0:57:59.480
Question: So, is there some sort of[br]dedicated processor on that board,
0:57:59.480,0:58:02.390
or do you use like a Microblaze[br]in the FPGA?
0:58:02.390,0:58:06.810
Tim: We use an open source soft core.[br]One of three.
0:58:06.810,0:58:11.820
We can change which soft core[br]we’re using with a command line flag.
0:58:11.820,0:58:16.390
We’re using either the LatticeMico32
0:58:16.390,0:58:19.790
which is produced[br]by Lattice Semiconductor.
0:58:19.790,0:58:24.550
We can use the OpenRISC-1000
0:58:24.550,0:58:28.530
or we can use a RISC-V processor.
0:58:28.530,0:58:34.750
We generally default to LM32[br]because it’s the most performance
0:58:34.750,0:58:40.430
for least FPGA resource trade-off.
0:58:40.430,0:58:46.240
But if you like RISC-V[br]or OpenRISC-1000 better
0:58:46.240,0:58:51.920
for some reason, say, you want[br]to run Linux on our soft core,
0:58:51.920,0:58:58.450
then you can do that. With a one line[br]command line change, yeah!
0:58:58.450,0:59:03.730
We’re looking at adding[br]J-Core support in early next year.
0:59:03.730,0:59:07.910
J-Core is quite big, though,[br]compared to LM32. So,
0:59:07.910,0:59:11.520
it probably won’t fit on some[br]of the very small devices.
0:59:11.520,0:59:14.270
Question: So it’s a Lattice FPGA?
0:59:14.270,0:59:21.250
Tim: It’s a Spartan6 FPGA. And our new[br]boards will probably be Artix-7
0:59:21.250,0:59:24.180
But we’re still in the process[br]of making them exist yet.
0:59:24.180,0:59:25.660
Question: Thanks.[br]Tim: I’ve also been working with
0:59:25.660,0:59:30.290
bunnie’s NeTV2, porting[br]our firmware to that,
0:59:30.290,0:59:33.460
which has been really awesome.[br]He’s doing some cool work there,
0:59:33.460,0:59:39.910
and he’s kind of inspired this whole[br]development by showing that,
0:59:39.910,0:59:43.420
yes, you could do this,[br]and you shouldn’t be scared of it.
0:59:43.420,0:59:45.820
Herald: Good, one more question[br]from microphone number 1.
0:59:45.820,0:59:53.880
Question: Yes. Do you have any plans for[br]incorporating HD-SDI into your platform?
0:59:53.880,0:59:58.560
Tim: Yes and no![br]We have plans and ideas
0:59:58.560,1:00:01.650
that we could do it
1:00:01.650,1:00:07.330
but HD-SDI
1:00:07.330,1:00:13.230
and all of the SDI protocols are[br]much harder for the consumer,
1:00:13.230,1:00:17.330
generally to access, and we want[br]to drive the costs of this down
1:00:17.330,1:00:22.070
to as low as it can go. And…
1:00:22.070,1:00:26.060
HDMI is a consumer electronic thing.[br]You get it on everything.
1:00:26.060,1:00:30.170
You get it on your[br]like five-buck Raspberry Pi.
1:00:30.170,1:00:35.120
HDMI is probably[br]a really good solution for this.
1:00:35.120,1:00:38.190
We haven’t developed any SDI cores[br]or anything like that,
1:00:38.190,1:00:43.330
so I can’t tell you like[br]that we’re doing anything there
1:00:43.330,1:00:49.460
but if somebody’s interested, again,[br]I like to remove roll (?) blocks and
1:00:49.460,1:00:53.230
we would love to have people work on that.
1:00:53.230,1:00:56.260
Herald: We have one more question from[br]the internet and we have two minutes left.
1:00:56.260,1:01:01.810
Signal Angel: OK, thank you. The question[br]is not related to HDMI but to FPGAs.
1:01:01.810,1:01:06.170
FPGAs are programmed in a high level[br]language like VERILOG or…
1:01:06.170,1:01:11.030
after simulation you compile. So every[br]vendor has created his own compiler
1:01:11.030,1:01:15.990
for its own hardware. Are you aware of[br]a move to open source compilers
1:01:15.990,1:01:21.220
or to independent hardware? And do you see[br]a benefit in open source FPGA compilers?
1:01:21.220,1:01:27.450
Tim: Yes! If anybody knows
1:01:27.450,1:01:30.590
about FPGAs you know[br]they use proprietary compilers.
1:01:30.590,1:01:35.510
And these proprietary compilers[br]are terrible.
1:01:35.510,1:01:39.560
I’m a software engineer.[br]If I find a bug in gcc
1:01:39.560,1:01:43.750
I can fix the bug. I’ve got those skills,[br]and I can move forward or at least
1:01:43.750,1:01:47.220
figure out why the hell the bug occurred.
1:01:47.220,1:01:51.600
That is not the case with FPGA compilers.[br]The FPGA compiler we use
1:01:51.600,1:01:56.970
is non-deterministic. You can give it[br]the same source code and it produces
1:01:56.970,1:02:01.800
different output. I’d love somebody[br]to reverse-engineer why that occurs
1:02:01.800,1:02:07.470
because I’ve removed all the randomness[br]from random sources from it
1:02:07.470,1:02:11.890
and it still manages to do it![br]I’m really impressed. So,
1:02:11.890,1:02:16.940
Clifford has done an open source[br]FPGA tool chain
1:02:16.940,1:02:21.980
for the Lattice iCEstick things.
1:02:21.980,1:02:28.940
He said he’s gonna work[br]on the Actrix7 FPGAs.
1:02:28.940,1:02:34.180
Please donate to him and help him.[br]I would… like…
1:02:34.180,1:02:38.030
if that exists I owe people who…[br]like a bazillion of beers, because
1:02:38.030,1:02:43.270
the sooner I can get off proprietary[br]tool chains the happier I will be,
1:02:43.270,1:02:48.730
and it will make my hobby so much nicer.[br]So, please help him!
1:02:48.730,1:02:50.930
Herald: And do give Tim[br]a big round of applause!
1:02:50.930,1:02:54.510
applause
1:02:54.510,1:02:58.300
postroll music
1:02:58.300,1:03:18.501
subtitles created by c3subtitles.de[br]in the year 2017. Join, and help us!