1
00:00:00,000 --> 00:00:15,520
wikipaka preroll music
2
00:00:15,520 --> 00:00:22,720
Hello and welcome to my talk on Remixing
Linux. My name is The one with the braid,
3
00:00:22,720 --> 00:00:30,480
and today I'm going to introduce your own
creation of a Linux distribution. We will
4
00:00:30,480 --> 00:00:37,280
talk on remixing Linux, on the
architecture you require to run an own
5
00:00:37,280 --> 00:00:50,830
Linux distribution, and some of the usual
use cases. OK, let's start. Well, what are
6
00:00:50,830 --> 00:00:57,356
use cases, what are purposes of your own
Linux distribution? Of course, you could
7
00:00:57,356 --> 00:01:02,544
customize Linux distribution for your
enterprise environment. For example, if
8
00:01:02,544 --> 00:01:08,152
you require special network settings, your
own services to be installed, it's
9
00:01:08,152 --> 00:01:15,298
sometimes easier to create an own
distribution than using a ready to use
10
00:01:15,298 --> 00:01:21,280
distribution and customizing it. Another
option is that you discovered something
11
00:01:21,280 --> 00:01:27,885
fully new and decide to create your own
Linux distribution. That's what we call
12
00:01:27,885 --> 00:01:34,015
yet another Linux distribution.
Please, please do not do this. We already
13
00:01:34,015 --> 00:01:42,293
have enough different flavors of Linux.
OK, another valid reason is missing
14
00:01:42,293 --> 00:01:46,640
software. If you require a special
software for use case and you want to
15
00:01:46,640 --> 00:01:52,450
bundle it into your operating system.
That's a usual use case. Moreover,
16
00:01:52,450 --> 00:02:02,251
embedding for microcontrollers is a common
use case for Linux remixing, as well as
17
00:02:02,251 --> 00:02:11,204
OEM hardware support for computer vendors
to bundle Linux. Some known vendors do
18
00:02:11,204 --> 00:02:18,509
this. Yeah, and of course, another desktop
or in general different preinstalled
19
00:02:18,509 --> 00:02:24,760
applications are a good reason to create
your own distribution. OK, what are we
20
00:02:24,760 --> 00:02:30,624
going to have a look on today? We will
look on live systems, because that's the
21
00:02:30,624 --> 00:02:36,394
most common use case. Afterwards we will
go through step by step instructions for
22
00:02:36,394 --> 00:02:44,131
and next, we take an example of Ubuntu. We
talk on making your system installable and
23
00:02:44,131 --> 00:02:49,855
we will talk about a simple tool
simplifying some of these steps. We will
24
00:02:49,855 --> 00:02:55,370
have a look on Linux from scratch, meaning
compiling everything yourself without
25
00:02:55,370 --> 00:03:01,320
using any templates or anything. And we'll
look at the architecture you require to
26
00:03:01,320 --> 00:03:07,354
run your own Linux distribution, a.k.a.
package mirrors. OK, let's start with a
27
00:03:07,354 --> 00:03:15,080
live system. What are properties of a live
system? Of course it needs to fulfill some
28
00:03:15,080 --> 00:03:22,760
requirements. If you use a tiny pen drive,
I don't know, eight gigabyte or 16
29
00:03:22,760 --> 00:03:30,240
gigabyte, it's obvious you need to
compress your whole system. Moreover, it
30
00:03:30,240 --> 00:03:37,022
must be writable at runtime. Just try to
mount your root filesystem in Read-Only and
31
00:03:37,022 --> 00:03:43,120
run a web browser. You run into big
trouble. Same for network connections. If
32
00:03:43,120 --> 00:03:50,248
you connect to wifi without writable root
system, you will run into big trouble. And
33
00:03:50,248 --> 00:03:55,967
another requirement, it's not really a
requirement, but it's recommended is an
34
00:03:55,967 --> 00:04:00,844
accessible live media. So if we have our
pen drive containing our live Linux
35
00:04:00,844 --> 00:04:05,683
distribution, we sometimes need to access
the pen drive itself in our live
36
00:04:05,683 --> 00:04:12,113
environment. For example, if we'll have
packages of our distribution inside or on
37
00:04:12,113 --> 00:04:19,668
our pen drive, we may want to install them
later. Then it's quite good to have your
38
00:04:19,668 --> 00:04:26,880
live media accessible. Well, of course, we
could script everything on our own, but
39
00:04:26,880 --> 00:04:33,677
someone already did all these steps and
has put them into a tiny software called
40
00:04:33,677 --> 00:04:40,600
Casper. Casper does exactly what we want,
creates a union file system, meaning an
41
00:04:40,600 --> 00:04:48,335
overlay on the existing file system and
creates a kind of fake writable system in
42
00:04:48,335 --> 00:04:53,941
memory, file system in memory. So it
allows modification of files, creation of
43
00:04:53,941 --> 00:04:59,422
new files while they are actually being
stored in the memory instead of on our
44
00:04:59,422 --> 00:05:05,720
physical disk. It mounts the live media
and it shows up a pretty display manager
45
00:05:05,720 --> 00:05:12,035
you can see over here. That's the Ubuntu
mate, it asks whether you want to try our
46
00:05:12,035 --> 00:05:19,160
system or whether we want to install our
system. OK, let's talk about the
47
00:05:19,160 --> 00:05:24,935
compression, because the compression we
require is tremendous. If we have a normal
48
00:05:24,935 --> 00:05:32,546
desktop distribution. Believe me, it
consists of about 30 gigabyte of data.
49
00:05:32,546 --> 00:05:37,814
Our applications, as soon as you install
some more applications like LibreOffice or
50
00:05:37,814 --> 00:05:46,773
a Web browser and some tools, you will
reach the 30 gigabyte or 20 gigabytes, but
51
00:05:46,773 --> 00:05:53,280
if you ever downloaded a live
distribution, you will realize it's not 30
52
00:05:53,280 --> 00:06:03,680
gigabyte. It's about two gigabyte or one
point five. I noted in my slides. Yeah, it
53
00:06:03,680 --> 00:06:10,560
means we need to compress our file system
on about 5 or 10 percent of the actual
54
00:06:10,560 --> 00:06:19,200
size. That's not difficult. There are many
compression algorithms we may use, but we
55
00:06:19,200 --> 00:06:26,640
have another requirement. We need
performant readability. As soon as we boot
56
00:06:26,640 --> 00:06:33,280
a system, our kernel won't stop accessing
any files, reading files, writing files.
57
00:06:33,280 --> 00:06:38,890
So we need a performant readability
because no one wants to wait five minutes
58
00:06:38,890 --> 00:06:45,840
to see the Web browser. And that's what
the squash file system is made for. The
59
00:06:45,840 --> 00:06:52,080
squash file system is a compressed file system.
It's kind of a disk image, but more
60
00:06:52,080 --> 00:06:59,440
advanced. It's used in diverse Linuxes.
It's used in the operating system of
61
00:06:59,440 --> 00:07:05,840
Chromecast and in the Linux userland it's
used by app images. These are these click-
62
00:07:05,840 --> 00:07:12,960
to-run Linux binaries you can download,
and it's used by snap packages from
63
00:07:12,960 --> 00:07:19,840
canonical's proprietary snap package
manager, which is unfortunately very
64
00:07:19,840 --> 00:07:27,200
widely spread. Yeah. Now we know something
about compression. Let's talk about how
65
00:07:27,200 --> 00:07:33,600
the customization works. Well, first of
all, we download a template. We create a
66
00:07:33,600 --> 00:07:40,974
base system from which we start our
customization process. Well, we start
67
00:07:40,974 --> 00:07:48,925
the customization itself. We prepare a CD
environment. So when you put a bootloader
68
00:07:48,925 --> 00:07:56,104
on a CD, we pack our root file system we
just created, put it on the CD, put a
69
00:07:56,104 --> 00:08:05,491
kernel there, make it, harden it a bit,
and at the end we will pack it. OK, let's
70
00:08:05,491 --> 00:08:16,136
go through it, step by step. In this case,
we will have a look at Ubuntu, we will go
71
00:08:16,136 --> 00:08:23,150
step by step through the customization
process of Ubuntu. How do you start the
72
00:08:23,150 --> 00:08:31,393
customization process? Well, you need an
existing ISO file from a live environment
73
00:08:31,393 --> 00:08:38,782
of Ubuntu, because it contains the
bootloader, the directory structure and
74
00:08:38,782 --> 00:08:47,942
everything, all the assets. Of course, you
could create them on your own, but, well,
75
00:08:47,942 --> 00:08:54,720
it's more work than you have to do. You
can easily use the existing bootloader and
76
00:08:54,720 --> 00:09:00,522
the existing files and directory structure
from an existing ISO file. Well, let's
77
00:09:00,522 --> 00:09:07,440
extract it and we're fine. We're done with
it. Afterwards, we need to create a base
78
00:09:07,440 --> 00:09:14,547
system. A base system is, well, the base
of Linux. We have the base utils, the
79
00:09:14,547 --> 00:09:24,589
base files, the directory structure, the
kernel, a shell. The package manager of
80
00:09:24,589 --> 00:09:32,670
our operating system we have. Without any
unnecessary tools, without, just the base
81
00:09:32,670 --> 00:09:40,988
of everything. You can either download the
prepared base system of the distribution for
82
00:09:40,988 --> 00:09:48,560
download, but you could create one on your
own using tools like the other Debian tool
83
00:09:48,560 --> 00:09:55,122
to create base system or the corresponding
tools from other distributions. Yeah, as
84
00:09:55,122 --> 00:10:01,356
soon as you have your base system, you
need to change your root inside this base
85
00:10:01,356 --> 00:10:08,696
system. And then you start the
customization process by installing the
86
00:10:08,696 --> 00:10:15,412
necessary drivers. We already talked about
Casper. Casper is this tool providing the
87
00:10:15,412 --> 00:10:23,542
live software and the live scripts, the
overlay file system in Ubuntu based
88
00:10:23,542 --> 00:10:31,641
systems. And it creates an initial RAM-
file-system and installes the required
89
00:10:31,641 --> 00:10:37,876
kernel modules to load and to boot a
squashed file system. So let's install
90
00:10:37,876 --> 00:10:45,909
these three packages, Casper, Ubiquity-
Casper and Lupin-Casper. And afterwards we
91
00:10:45,909 --> 00:10:52,160
can install whatever we want. We can
install our loved vim, we can install our
92
00:10:52,160 --> 00:10:59,595
loved Alpaka software, and we can install
an alternative shell, the packages we
93
00:10:59,595 --> 00:11:07,220
want. In the example code I provided here
we look on changing the plymouth screen.
94
00:11:07,220 --> 00:11:13,017
That's the startup screen which shows up
when you boot the distribution. I don't
95
00:11:13,017 --> 00:11:19,610
know, you could put an Alpaka in there or
the logo of rC3. Yeah, that's what we do
96
00:11:19,610 --> 00:11:27,280
in customization or modification of the
system. Afterwards, we continue packing
97
00:11:27,280 --> 00:11:33,654
everything we created. First of all, we
need a Kernel to boot , because well
98
00:11:33,654 --> 00:11:39,790
before our base system can be extracted,
we need to load a kernel by the bootloader
99
00:11:39,790 --> 00:11:46,687
and the kernel must contain the
corresponding kernel modules to load, to
100
00:11:46,687 --> 00:11:52,888
boot a squash file system. That's what
Caspar already prepared for us, we only
101
00:11:52,888 --> 00:12:00,496
need to create the initial run file system
and if you read the code I provided, you
102
00:12:00,496 --> 00:12:08,037
will see I was missing an minus between
update and init RAM file system. Let's
103
00:12:08,037 --> 00:12:16,558
create our initial RAM file system and
copy them or copy it together with our
104
00:12:16,558 --> 00:12:25,250
kernel to our extracted live environment.
And can delete the prior kernels and so
105
00:12:25,250 --> 00:12:30,600
on. We don't need them. We have our own
kernels now. Afterwards we pack our base
106
00:12:30,600 --> 00:12:36,315
system. Of course we need to unmount any
mount points in the base system we created
107
00:12:36,315 --> 00:12:42,943
and create a squash file system from it.
And these three files, the VM Linux, the
108
00:12:42,943 --> 00:12:50,120
initial Ramdisk and the squashfs file
system, they are the major pillars of our
109
00:12:50,120 --> 00:12:55,680
live environment. If we have those three,
we are actually ready to go now. We can
110
00:12:55,680 --> 00:13:00,990
either use our existing bootloader from
our extracted ISO file or we, of course,
111
00:13:00,990 --> 00:13:06,961
you can install your own bootloader in
this directory. But well, it's easier to
112
00:13:06,961 --> 00:13:13,874
use the existing. For security purposes
and for hardening your live environment I
113
00:13:13,874 --> 00:13:22,148
would recommend to put the MD5 sums of all
files in your ISO file in this
114
00:13:22,148 --> 00:13:28,375
environment, because as soon as you, for
example, copy your distribution, for
115
00:13:28,375 --> 00:13:36,934
example, on a pen drive on which is fat or
ext4 formatted, it could be modified by
116
00:13:36,934 --> 00:13:43,430
anyone. And that's why it's much better to
verify the MD5 sums at the boot process.
117
00:13:43,430 --> 00:13:50,379
Casper will automatically detect whether
the MD5 sum, the txt file exists and if
118
00:13:50,379 --> 00:13:56,564
it exists, it will automatically verify
these MD5 sums. OK, now we are actually
119
00:13:56,564 --> 00:14:03,382
ready to go, ready to pack up our ISO
file. There's a famous tool called
120
00:14:03,382 --> 00:14:11,029
genisoimage. So generate an ISO image,
that's a command you could use. The
121
00:14:11,029 --> 00:14:20,080
command takes care of legacy booting
except of output file, the disk and the
122
00:14:20,080 --> 00:14:28,400
disk image and the -r parameter, you could
leave out everything if you do not require
123
00:14:28,400 --> 00:14:37,320
legacy bootloaders. But if you want to
obey legacy bootloaders, you should also
124
00:14:37,320 --> 00:14:44,960
hybritize your ISO file to be able to dd
the ISO file to a pen drive. Otherwise,
125
00:14:44,960 --> 00:14:51,600
without the ISO hybrid command, it won't
boot from, it would boot from a CD, but
126
00:14:51,600 --> 00:15:01,280
not from a pen drive. Yeah. OK, now we
created our own Linux distribution. Well
127
00:15:01,280 --> 00:15:05,360
not that much. It's actually just a
repacked Linux distribution. Now we talk
128
00:15:05,360 --> 00:15:11,040
about making it a real distribution.
That's why we talk about making it
129
00:15:11,040 --> 00:15:18,560
installable. Why should we make it
installable? Of course we could manually
130
00:15:18,560 --> 00:15:25,280
install our system. We could manually format
our hard drive, extract our file system and
131
00:15:25,280 --> 00:15:32,960
manually create users and so on, et
cetera, et cetera. But most of users
132
00:15:32,960 --> 00:15:39,920
prefer graphical way except of these geeks
taking part at the rC3. That's why an
133
00:15:39,920 --> 00:15:48,400
installer may be useful. An installer does
exactly what we need. It creates
134
00:15:48,400 --> 00:15:54,560
partitions, creates users, installs
language-packs, offers installation of
135
00:15:54,560 --> 00:15:59,760
extra software and does everything the
user might want. Creates users, connects
136
00:15:59,760 --> 00:16:05,760
to wifi, whatever. And there are three
very common installer frameworks:
137
00:16:05,760 --> 00:16:12,080
Ubiquity, Calamaris and Anaconda. Now
we'll have a look at these frameworks.
138
00:16:12,960 --> 00:16:21,760
Ubiquity is known from Ubuntu. It's ready
to use if installed, but it's actually not
139
00:16:21,760 --> 00:16:30,320
at all customizable. Anyway, some hackers
at GitHub found some ways to customize it
140
00:16:30,880 --> 00:16:36,320
and provided some good information, some
helpful information in their repositories.
141
00:16:36,320 --> 00:16:42,880
The next one is Calamaris. Calamiris is
highly customizable, is used by many
142
00:16:42,880 --> 00:16:50,720
distributions, including SUSE Linux, I
think Xubuntu, so the XFCE Ubuntu flavor,
143
00:16:51,440 --> 00:16:59,840
DeepinOS, I think I am unsure. I think
elementary as well. But if you only
144
00:16:59,840 --> 00:17:05,600
install it, it won't work. You need to
customize it. The third one is known from
145
00:17:05,600 --> 00:17:13,840
the Fedora project. It's called Anaconda.
It's a very advanced tool. It's able to
146
00:17:13,840 --> 00:17:19,520
update existing systems. If we have an old
Fedora installation, we can boot our live
147
00:17:19,520 --> 00:17:26,640
CD and update the existing installation
and moreover, very useful for admins is
148
00:17:26,640 --> 00:17:32,640
the remote functionality. It can be
started via SSH and automatically update
149
00:17:32,640 --> 00:17:38,400
everything and automatically install and
deploy the software we need. Um, yeah.
150
00:17:40,240 --> 00:17:46,720
Well, I already mentioned, for everything
you can manually do and customize, there
151
00:17:46,720 --> 00:17:52,080
is a graphical tool. If you prefer the
graphical way you could use the tool
152
00:17:52,080 --> 00:18:00,560
remastersys, aka LinuxRespin. That's the
the older name. It offers a simple
153
00:18:00,560 --> 00:18:07,120
graphical user interface for packing your
currently installed system into a live
154
00:18:07,120 --> 00:18:13,040
distribution, allowing to install it. It
does not allow a high amount of
155
00:18:13,040 --> 00:18:16,880
customization in the graphical way, of
course, you can manually perform it on the
156
00:18:16,880 --> 00:18:24,080
command prompt. But it's an easy way to,
for example, backup your just installed
157
00:18:24,080 --> 00:18:29,920
system or you readily configured system to
a pen drive to make it installable later.
158
00:18:34,160 --> 00:18:40,480
And now, Linux from scratch. I guess most
of you already heard of Linux from scratch
159
00:18:40,480 --> 00:18:45,920
from scratch means, yeah, pretty much
Linux from scratch. You start without
160
00:18:45,920 --> 00:18:51,680
anything. You start on your host system
and create an own system from scratch.
161
00:18:53,120 --> 00:18:59,120
What does that mean? It means you should
have lots of time. Lots of RAM, and many
162
00:18:59,120 --> 00:19:05,120
processor cores. Because you need to
compile everything yourself. Beside the
163
00:19:05,120 --> 00:19:10,480
kernel, you need to compile the whole
userland. If you want, you can include a
164
00:19:10,480 --> 00:19:15,840
package manager or whatever. But if you
make real Linux from scratch, you do not
165
00:19:15,840 --> 00:19:20,400
take any binaries but compile everything
yourself, everything on your own.
166
00:19:21,440 --> 00:19:28,720
Currently, minimally, you need to compile
226 dependencies. But in some years it
167
00:19:28,720 --> 00:19:36,000
will be some more dependencies and the
amount of dependencies required increases.
168
00:19:36,000 --> 00:19:40,800
Yeah, after you compile everything, of
course you need to configure everything,
169
00:19:40,800 --> 00:19:47,120
need to provide a bootloader. But as soon
as this is done, as soon as you created
170
00:19:47,760 --> 00:19:53,840
your own base system, your own system from
scratch inside a fake root, it's the same
171
00:19:53,840 --> 00:20:02,720
as we already talked on during live system
customization. As soon as you have this
172
00:20:02,720 --> 00:20:08,080
fake root, you simply pack it to an image,
so a squash file system, write it on a
173
00:20:08,080 --> 00:20:13,840
disk, provide a bootloader, and that's it.
OK, one more word. I will talk about
174
00:20:14,560 --> 00:20:18,960
maintaining architecture for
distributions. I will talk about package
175
00:20:18,960 --> 00:20:24,880
servers. If you have a distribution, you
usually provide updates, security fixes
176
00:20:25,680 --> 00:20:32,960
and whatsoever. Maybe software the users
may install later. Yeah, that's a package
177
00:20:32,960 --> 00:20:40,320
mirror, a packet server. From Debian, you
can simply mirror existing servers, for
178
00:20:40,320 --> 00:20:47,920
example, the Debian or the Ubuntu or the
Linux Mint servers by using the tool apt-
179
00:20:47,920 --> 00:20:54,320
mirror. It simply mirrors, means it does
not allow any customization. For ArchLinux
180
00:20:54,320 --> 00:21:01,440
it's easy as well. You can simply use vftp
and crontab to download the packages and
181
00:21:01,440 --> 00:21:09,360
provide them on your host system, on your
server and synchronize from them. That's
182
00:21:09,360 --> 00:21:15,440
much easier than in the Debian
environment. But usually if you have a
183
00:21:15,440 --> 00:21:21,520
distribution, you want to customize it. At
Debian, the tool aptly is very widely
184
00:21:21,520 --> 00:21:26,080
spread because it allows very advanced
management of different snapshots from
185
00:21:26,080 --> 00:21:31,200
different repositories, mirrors, local
repositories, single packages, allows
186
00:21:31,200 --> 00:21:36,560
diffs and so on. But it does not allow any
optimization. So good luck at scripting on
187
00:21:36,560 --> 00:21:43,840
your own. If you look at ArchLinux, for
example, you would usually write your own
188
00:21:44,560 --> 00:21:50,320
package build files as you know them from
the Arch user repository and include the
189
00:21:50,880 --> 00:21:57,920
readily build packages into your directory
from which you serve your updates. That's
190
00:21:57,920 --> 00:22:02,080
much easier than in the Debian
environment. I still found one more
191
00:22:02,080 --> 00:22:08,400
graphic explaining aptly. That's what
aptly can do. It can do everything. Well,
192
00:22:08,400 --> 00:22:12,400
you have mirrors, you have local
repositories. You can take snapshots, take
193
00:22:12,400 --> 00:22:17,040
differences from snapshots, publish these
differences, publish merged snapshots of
194
00:22:17,040 --> 00:22:23,680
them, publish single snapshots, diff
everything and do whatever you want. It's
195
00:22:23,680 --> 00:22:30,640
very performant, the tool and yeah, but
it's hard to use. It requires some time to
196
00:22:30,640 --> 00:22:38,720
be learned. Yeah, that's it already on
remixing Linux. Now there should be an
197
00:22:38,720 --> 00:22:44,640
Q&A. Thank you very much for your
attention of this talk. I hope I could
198
00:22:44,640 --> 00:22:53,040
give you a good introduction into remixing
Linux and good luck at coding and enjoy
199
00:22:53,040 --> 00:22:56,904
your Remote Chaos Experience.
200
00:22:56,904 --> 00:23:00,731
Music
201
00:23:00,731 --> 00:23:13,160
Subtitles created by c3subtitles.de
in the year 2021. Join, and help us!