[Script Info] Title: [Events] Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I'm here today to talk to you about\Ndiffoscope Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and how you can use it as a better diff Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,or for Quality Assurance, etc., things\Nlike that. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Moin! Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Apparently that's like a north german\Nthing to say "welcome". Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,North german, north Denmark, Scandinavia,\Nthat kind of thing, I'm told. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,People are shaking their head, so I'm\Ngoing to assume that's true. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is my first PC, an IBM 5155. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Sometimes, when you rebooted it, it would\Nlaunch into, it would somehow revert Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,from booting from the hard disk to booting\Nfrom a basic ROM, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,as in the programming language ROM. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It was on my motherboard for some reason. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, randomly, you just get a chance to\Nprogram in basic and then, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,sometimes you wouldn't, I don't know why,\Nbut… yeah. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's quite fun with this kind of clicky\Nkeyboard, and that folded in Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and it was this kind of big desk thing. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Anyway… Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is my first Debian. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,At the time it was already old. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What's this one? Is this Slink? 2.2?\NYeah. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And this is when we had US and non-US,\Nso that's really dating if you remember that. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is my first contribution to Debian,\N19th December 2006, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,sending a patch to lillypond which is kind\Nof interesting Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and the response was "Oh yeah, rock on,\Nmany thanks. I'll upload this and Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,it'll be landing to Etch". Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And this was super motivating because\NEtch was just coming out and it was like Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,"Great, I've got let one line of tiny patch\Nin a release. This is super cool." Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Thomas' response was super motivating. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, after that, like that Christmas\Nbasically spent ??? Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Debian webpages and stuff. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Very well timed. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's kind of a good… Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You know, someone sends a patch, be like\N"Cool, thanks" Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Like a little notice in the changelog. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It was, you know, so stupid but…\NYeah, do that kind of thing. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, moving on. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Why diffoscope?\NWhy did we write diffoscope? Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What's the background here? Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It comes from reproducible builds. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The very quick outline is that once you\Nget the source code for free software, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you download the source code for nginx\Nor whatever, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,pretty much everyone just runs binaries\Non their servers or their systems. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You know, "apt install bla", "yum install",\Nwhatever. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Android Playstore, whatever. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Can you actually trust whether these two\Nthings correspond with each other? Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You've gotten the source code, it looks\Nalright, and then you install this binary, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,yeah… Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Who generated that? Can you trust that\Nprocess? Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Can you trust who generated it? Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Even if you could trust them, could you\Ntrust them not to be exploited? Etc. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is a big problem because you can\Nexploit a build farm and then Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,obviously exploit all of that, you know,\Na trojan into the build farm, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,so every single binary that comes out\Nis compromised. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Kind of problematic. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You could also target individual developers\Nmachines, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,so I could go of to, say, your machine,\Nadd a backdoor to it, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,so every binary that you give to friends\Nand things like that, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,are compromised in some way, stealing\Nyour bitcoins or whatever. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I can also ???\Nand blackmail you into producing Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,software that has compromises or extra\Nfeatures, shall we say, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,that don't exist in the source code. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So what will happen there is that you'd\Nrelease your source Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and the binaries you produce have\Nthis sort of backdoor that, you know, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,someone is forcing you into producing. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, you don't want to do that. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Anyway Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,enough of that. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What you do for reproducible builds is you\Nensure that every time you build Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,a piece of software, you get an identical\Nresult. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Multiple people then compare their builds\Nand check whether they all get Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,the same results Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and this means that an attacker must\Neither have infected everyone Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,at the same time, or they haven't\Ninfected anyone. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The point here is that you have to ensure\Nthat builds have identical results. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Ok, great. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, we started the reproducible builds\Nproject, etc. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And we build 2 debs. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Oh, I'm sorry about the colors there. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You probably can't see that. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That says "sha1sum a.deb b.deb". Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Anyway, we're comparing the sha1sums\Nof 2 binary Debian files. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, these two files differ. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Ok, they're not reproducible. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Why is that? Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So we run a diff on them. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Yeah… Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, what can we learn from this? Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Well, not very much, visibly they're\Ncompressed so Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,as soon as we see one change, we'll see\Nthey would just cascade changes Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,because that's how compression works. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I guess we know it's a deb ???\Nformat file, not very useful. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Ok, great so we're gonna have a look in Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We'll do a binary diff and ok, well… Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Again, that's not really telling us\Nvery much Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,with the diff there. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Ok, great. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,??? Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,"ar x" is on the new maintainer thing,\N"how you unpack a deb" Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Everyone remembers this, right? Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You unpack a.deb with "ar x" and you\Ndo that to b.deb Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and then we diff the results of that. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Ok, so…yeah, 7zip. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Ok, compressed content, not very useful. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Ok, so let's unpack the control.tar inside\Nthese debs. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And then we run diff on that. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Still not really telling anything useful\Nabout how to make this package reproducible Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So let's unpack the tar.xz into the tar. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Inside that tar, there's a file called\Nmd5sums and we start to see some differences Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,between some files in these two debs. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,??? meaningful, so now\Nwe have some idea that Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,it has something to do with this\Nusr/bin/pmixer binary. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Ok, interesting. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We'll unzip that and then we do a diff on\Npmixer itself. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now we're back into just binary\N??? mode Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This isn't very helpful and this is taking\Nquite a while Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and if I remember correctly, Debian has\Na lot of packages. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So this might take a little while. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, basically, ??? meme Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I should build a better diff. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's not quite true, this is actually… Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It was lunar that started this project Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and it was called debbindiff, because\Nwe wanted to diff Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,binary Debian packages. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So this is the initial commit, 2014. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,"The version is successfully able to report\Ndifferences in two .changes files. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Not with much interesting details,\Nbut it's a start." Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And it was a start. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Fast forwarding… Oh, sorry about these\Ncolors, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I don't know if we can do anything about\Nthe lights? Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Yeah? Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,No? Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Alright, well… Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Basically, we're diffoscoping on… Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It works kind of diff does normally, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you give it two files, it outputs\Na unified diff. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So "diffoscope a b", one file contains\Nthe word "foo", one contains the word "bar". Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Nothing actually out of the ordinary. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's sort of colored by default, so that's\Nwhy you can't see it, but whatever. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It supports archive formats, so if you\Ngive it two tar files, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,if we then tar up our "a" file and\Nour "b" file into a a.tar and b.tar Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and then run diffoscope on those tar files Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,we get this kind of, like, hierarchy here. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So it's saying that there are differencies\Nbetween these files, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,in the file list they have different time\Nstamps, because I made them Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,at different times, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and here are the contents, so we got\N"foo" there and "bar" there. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So we can see the difference between them. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Well, I can, I don't know if you can,\Nyou get the slide there. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If we gzip these tar files and then run\Ndiffoscope on those gzip things, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,it'll say "ok, what we've done is unpack it\Nfirst, and here's the metadata Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,about the gzip process", Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and inside that are a.tar and b.tar\Nfrom the previous slides. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And then the "a" file and the "b" file. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, it's really going two levels deep\Ninto this tar.gz file. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's pretty cool. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And it's completely recursive, I think\Nit will actually blow out after, I think, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,1000 [levels]. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,[light is turned down for the audience\Nto see the slides] Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I'll just bump back a bit, just in case. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,[Applause] Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Thank you. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So that's the a and b files. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We've tared them up and so I see\Nthe hierarchy of foo and bar file layer. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I've gziped them, so this is a gzip layer. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Here's the tar layer and then there's\Nthe files themselves. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is from a real .deb from the archive. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Inside this .deb, there's a data.tar.xz\Nand in that xz file there's a data.tar Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and inside that tar file, there's a file\Ncalled aff and inside that Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,there's a version string that is different. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And that looks like a build date so we\Nprobably know that if we went back Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,to the source package, we could very\Nquickly work out, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,with get a very quick grep, work out\Nwhere this file is being generated from, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,the de_DE.aff file and then ???\Nprobably quite obvious Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,that it's using the current build time\Nand then we can just patch that, fix it etc. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is gone from two rather obscure\Nbinary .debs all the way to the fix Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,probably in about 5 minutes, and you can\Nprobably send the patch in that time Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,because it'd be quite quick. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Without diffoscope here, without this sort\Nof recursive unpacking, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you'd be just completely lost, you'd be\Nthere with arx all day Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and working out which files are different\Nand trying to use xxd Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and this kind of nonsense. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,diffoscope's got some other things as well Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,if you try to do reproducible packages\Nand things are varying just on Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,the line ordering, we detect whether\Na file differs only in the line ordering. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, here's file "a", "These lines are in\Norder". Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,File "b" has "These order are in lines". Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's very difficult to say, actually,\Nit's like one of these tongue twisters. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Run diffoscope on these two and it says\Nit's got ordering differences only. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's interesting, so you probably need\Nto sort, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you go all the way back to the source code,\Nwork out very quickly, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,if you know it's just ordering differences\Nyou just kind of know Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,what the output's gonna be, you can\Nsearch for order in ??? Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and you get the right files, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,??? sort in the right place,\NBAM, send it patch of (???), Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,everything is great. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Oh, and send it to upstream as well\Nbecause you're good. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It supports a lot more things. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We've been showing the terminal\Ntext output here. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's got a HTML output mode, which is\Nreally useful in the hierarchal thing Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,when it gets a bit more complicated. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Instead of being laid on top of each other\Nlike a unified diff, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you get the diff on the left and the right\Nand you get sort of a nested Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,thing inside with colors and lines and\Nyou can link this and various things in it Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,including bits of metadata here, other\Nbits here, what command you used. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's the HTML output. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We also support a lot of file formats,\Nit's not just on text, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,it's about all of these, so let's quickly\Nrun through some of them. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You give it two Androip apk files which\Nare kind of like zips, but magic. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It'll know how to compare them. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There's like a Manifest file that needs\Ndecoding. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It supports Berkeley DB databases, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Word documents, that's a Word document\Nwith "a" and that's a Word document with "b" Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and it'll correctly do that. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you run that through diff normally,\Nthat ??? be a binaly mess, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,so completely useless. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,E-books, there's epub, it also supports\Nmobi. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So if you give it two epub files, it'll say\N"They just differ in this date". Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Brilliant. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Normally that will be completely useless\Ndiff binary ??? Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So you can be like "epub date, ok", grep\Nthe source code for that, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,make a patch really quickly. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Mono binaries, git repositories, why not? Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Gnumeric spreadsheets, ISO images. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Oh yeah, ISO images is really cool. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, it'll basically unpack the ISO, then\Ninside that there might be a squashfs image Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,then it'll completely go down to that and\Nwork out any differences Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,between the two contents in the ISO file,\Nincluding any metadata. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is on the squashfs metadata headers,\NI think. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But say inside that ISO, there was a file\Nthat was a pdf, and inside that pdf was Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,a ??? which varied, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,it will basically go all the way down\Nand say "yeah, it's actually here, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,in this ??? that the data differs." Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And that means you can just go again\Nall the way back to the source Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and say "ok, cool, we know how to fix\Nthis quite quickly" Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And this is really valuable in getting\Nthe recent Tails distribution reproducible Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,so their ISOs are reproducible. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you build one and I build one, we get\Nthe exact same one Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and that's kind of useful for something\Nlike Tails where you would probably want to Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,of all, there's a lot of projects that you\Nmight want to compromise, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you might want to go after that one,\Nbecause of the kind of people that are using it. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We support comparing images, so this is\Nusing ??? Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and then just running that through diff. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That is a linux penguin and that is\Nsomething else, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I can't remember now. Oh, FT. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It supports images. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It supports JSON and pretty print,\Nso if you give it two JSON files Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,one with key/value… it'll do a nice\Ndiff of them. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It will pretty print it first, before\Ndoing the diff, so it'll actually give you Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,something clean, otherwise I don't know\Nif you've ever diffed Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,two very long JSON lines, if they differ\Nin the middle, you just get Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,a huge long unified diff, but here it's\Nlike "oh, just ??? things have changed" Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,OpenDocument text formats,\NOgg audio files, because why not. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,tcpdump capture files, that's actually\Nquite useful. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,PDFs. That PDF says "Hello World" and\Nthis PDF says "Hello sick sad world", Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I don't know why. ???\Nin the demo. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Again, run that through normal diff\Nprogram… garbage. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,XML documents. Again, it'll pretty print\Nthem so it's nice, actually nice do read. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you want to get started on diffoscope,\Nthe very easiest and quickest way to do is Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,fire up a web browser, try.diffoscope.org,\Nselect your files, press Compare Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and it'll upload them and run diffoscope\Nwith all the support for all the file formats Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,in the cloud for you and give you a nice\NHTML page that you can then link to people Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So that's the very quickest way to get\Nstarted. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The next quickest way is to install\Ntrydiffoscope and then you run that Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,on two files and it'll basically do\Nthe same thing, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,run it in the same cloud service as\Ntrydiffoscope Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,but it'll give you the result on the\Ncommand line or Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,if you pass the webbrowser option, it will\Ngive you an URL or load your webbrowser, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I can't remember exactly which, with\Nthe same results. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is 1kB of Python, nothing basically. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's the next easiest way. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But you can then install diffoscope itself\Non your own machine. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I recommend not installing recommends\Nbecause all of those file formats Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,might drag in extra things about\Nthe whole of TeX, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I think the whole of OpenOffice, whole\Nof Mono, whole Java… Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Android, yeah, quite big. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I think there's another big one I can't\Nthink of. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,They're all optional, and they all say\N"By the way, I support TeX documents Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,or whatever, Mono, whatever. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But you need to install this package and\Nthen you get full pretty printed support", Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And it'll tell you that when it's missing. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, if you just start with\N--install-recommends disabled, Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,right on your file, if it says\N"please install this package, you can then Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,install them as you go along, as you want" Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,rather than installing everything. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And then ??? and then works as before Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I you can improve all your own quality\Nassurance and debian packaging Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,with different scope Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The biggest value here is not\Nnecessary for reproducible builds Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's for basically just seeing where you\Ndo want to have a diff or expecting a diff Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and you are expecting a particularly type\Nof diff in a particularly way Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you can basically see those changes\N Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And if you build two debs normally and\N... i'll try to demo in a second Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You build a deb with a patch applied you\Ncan ??? see a diff on the source package Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But that's not very useful because the\Nbinaries are going to end in the Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,people machines. But if you run a diff on\Nthe binary itself, did that change and Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,really hit the binary, I think really ...\NNo.. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I just run through a very live demo of\Ncourse, so it's gonna fail ... Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Checkout some .... We'll get this \Nlibnetx-java Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We just build that once Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Lets say we are on security team and Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,want to apply a patch, and we want to be\Nreally sure because we are to push it out Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,to all our users Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,First we will make a changelog Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Closing a bug Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Find some java file to change Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's pretend we have a real patch Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's replace that equals equals,\Nsay that was the fix Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So that's the patch from upstream Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Upstream blast patch Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,When we build this what we wanna see is\Njust that change in the file Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,we wanna see any nonsense changes of \Nextended ??? but we also definitely want Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,to see that change, cause if our binary as\Nfor security reasons don't have that change Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,the we aren't fixing people machines,\Nthey will issue a DSA ??? installed, saying Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And you should do proper testing as well\Nat multiple levels Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I will build that again Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So we wanna diff the original one Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We wanna diff that one with a fake \Nsecurity one Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You see on the progress bar 100%\N1- there are diferences (there should be Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,diferences)\NLets see what that diferences are Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,in our web browser, its a nice html output Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let have a look.\NAre we seeing what we wanna see? Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There are some chances in the data ta, we\Nkind of expect that Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Whats changed in our control file?\NWell the version changed,we wanted that Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,to change. Perfect Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And its changed to ???\NThat's what we wanna see Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,No other changes here so there was no \Nweird control or in magic going on Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,In our data tar the color of the timestamp\Nchanges, we will ignore these for now Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The changelog has changed, well I hope so\Nbecause I have changed that entry\N Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Here is where we going to start seeing\NWe are going to see the changing in the Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,jar file which is the java class, java\Ncompile archive format Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We are seeing some meaningless timestamp\Nchanges but we can ignore those Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,??? cause its just metadata maybe Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Ok part of a class, so if you can see here\Nit's basically a de-compilation of the Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,java file itself and it's basically saying\N"oh I use to say if now and if not now"\N Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So these are the actual byte java\Nbyte code instructions and whats really Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And what is really ??? here\Nits that nothing else has changed\N Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We were just expecting that change between\Nthe two op codes, of if now elseif not not now Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,which is good cause its like it hasn't made\Nany code changes but also crucial we can Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,see that it has actually made a change\Nto the code. Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,For example its wasn't use some cached\Nversion or something like that Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is really useful Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And just running a naif diff wouldn't\Ngive that of course, because it would just Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,come with binary garbage\NAnd just seeing the diff had changed again Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,??? be told you anything, because all of the\Nchange would have changed as well Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So its like well yes it's diferent Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The meaningful change there it's\Nwhat actually fixes the "floor" Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,??? but we know it's there Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's kind of ??? \NShifting this deb out I'll be quite Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,confident, that this seemed like the\Nactual bug Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I've been quite confident pushing that out\Nbecause it's very minimal amount of changes Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you wanna do that for security reasons Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So this was the live demo Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The other one is seeing no changes\Nat all, so you can build once Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,if you build a reproducible Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can build once change your compiler\Nor change another part of your tool chain