0:00:05.885,0:00:12.871
Today, I talk about FAI.me, which is a[br]build server for images.

0:00:14.947,0:00:20.359
First, anybody that never heard anything[br]about FAI?

0:00:21.781,0:00:22.593
Ok

0:00:24.376,0:00:27.468
I started this project in 1999.

0:00:29.297,0:00:30.679
I'm not sure…

0:00:31.656,0:00:35.883
No, I'm sure that during those times, the[br]Debian installer did not have

0:00:35.883,0:00:39.703
the preseeding stuff, so we needed[br]something automatically.

0:00:42.751,0:00:50.347
I installed the first cluster with FAI and[br]I always do talks on FAI or

0:00:50.347,0:00:54.211
today in the lightning talks, I talk[br]a little bit about dracut,

0:00:54.211,0:00:55.794
which is used in FAI.

0:00:56.964,0:00:58.838
So, what was the motivation.

0:00:59.448,0:01:02.780
A neighbour of mine, she came to me with

0:01:02.780,0:01:08.109
"My Windows desktop is broken,[br]can you reinstall it?"

0:01:08.395,0:01:13.673
And in the end, I installed her Linux,[br]and I was shortly thinking about

0:01:13.673,0:01:18.223
"Should I use FAI for installing her[br]desktop with Linux?"

0:01:18.995,0:01:24.809
And in the end, I did not use it because[br]FAI is too complicated,

0:01:24.809,0:01:31.030
like the Debian installer, I guess it's[br]not really that easy for beginners

0:01:31.030,0:01:32.740
because there are a lot of questions

0:01:32.740,0:01:34.973
but also FAI is not really for beginners.

0:01:35.785,0:01:38.787
So this was the motivation about thinking[br]about FAI.

0:01:40.255,0:01:43.776
The target group was always advanced[br]sysadmins

0:01:43.776,0:01:49.313
but I thought maybe it's possible to make[br]FAI usable also for people

0:01:49.313,0:01:52.490
that are not that advanced sysadmins.

0:01:54.359,0:01:58.790
The idea is that an installer should cover[br]most installations.

0:01:59.075,0:02:02.890
The Debian installer is really perfect[br]because I think it covers

0:02:02.890,0:02:07.114
all different kinds and strange environments

0:02:07.114,0:02:14.146
You can do a lot of things, you can configure[br]very strange combination of language,

0:02:14.146,0:02:16.057
keyboard layout and so on

0:02:16.546,0:02:23.068
but I was thinking about an installer[br]that covers 90 or 95% of the installations

0:02:24.677,0:02:32.092
A lot of special cases can be ignored and[br]since the Debian installer has like

0:02:32.862,0:02:38.345
more than 20 questions, I thought it would[br]be much nicer if there were only

0:02:38.345,0:02:46.155
3 to 5 questions and I looked at Linux Mint[br]and Mageia installers, CentOS installer,

0:02:46.155,0:02:50.181
and they all ask much less questions.

0:02:51.767,0:02:57.408
In the Debian installer, we sometimes[br]have also things that are asked

0:02:57.408,0:03:01.512
during the installation, so not everything[br]is asked at the very beginning.

0:03:02.079,0:03:05.898
For example, the task selection, where you[br]select your desktop,

0:03:05.898,0:03:08.538
is done after the base installation.

0:03:08.904,0:03:13.130
This was also very important, I would like[br]to have something that

0:03:13.130,0:03:15.366
asks everything at the very beginning.

0:03:16.584,0:03:22.271
Then, maybe some tool could create[br]a customized installation image

0:03:22.271,0:03:27.149
and this installation image should run[br]then completely unattended

0:03:27.149,0:03:31.572
so you can get yourself a coffee and[br]when you come back, your machine is ready.

0:03:32.792,0:03:37.590
There are 3 things to customize installation[br]image,

0:03:37.590,0:03:43.440
you just put this image, you do not have[br]to touch anything, and then it's ready.

0:03:43.603,0:03:48.239
I thought "Oh yes, this is FAI, maybe[br]FAI can do this."

0:03:50.028,0:03:57.059
As I said, FAI is only, or was until now[br]only a tool for experienced sysadmins

0:03:57.059,0:04:02.297
and you have to adjust several config[br]files, these are ASCII files

0:04:02.297,0:04:08.677
but still you have to touch 5 to 10[br]config files to make a customization.

0:04:09.610,0:04:12.698
So, how can I make FAI usable for[br]beginners?

0:04:13.023,0:04:15.294
That's the beginning of FAI.me.

0:04:17.647,0:04:21.430
There's a web page, we'll show it[br]in more detail later,

0:04:21.430,0:04:26.831
where you can just click some things, and[br]then you get a customized image.

0:04:28.815,0:04:35.606
This image can be put onto a CD, DVD or[br]USB stick, just with dd

0:04:35.606,0:04:40.724
and the customization is just by using[br]the web interface

0:04:40.724,0:04:46.257
so there's no need for you to edit[br]a text file, a config file inside FAI.

0:04:48.333,0:04:53.454
I hope I covered most important things[br]that you want to adjust

0:04:53.454,0:04:55.852
or a little bit customize.

0:04:56.123,0:04:59.733
You can add additional packages, I think[br]that's the most important thing

0:04:59.733,0:05:03.752
that people say "I want to have the normal[br]Debian installation

0:05:03.752,0:05:05.778
but with some additional packages."

0:05:06.836,0:05:11.796
And you can select different[br]distributions, so it's not only

0:05:11.796,0:05:16.755
the installation image for the stable[br]release, you can create

0:05:16.755,0:05:19.190
3 variants of the installation.

0:05:21.553,0:05:28.747
This is the web page and thanks to Juri,[br]he did a great job

0:05:28.747,0:05:34.316
during the first and second day, he added[br]a new feature that we now have

0:05:34.316,0:05:35.616
a toggle button.

0:05:36.388,0:05:39.522
Is it big enough or should I zoom in?

0:05:40.823,0:05:41.633
Ok.

0:05:42.688,0:05:49.963
So, we have a toggle button, what you see[br]now is just the bare minimum of questions

0:05:49.963,0:05:54.111
and we can toggle it to more advanced[br]settings.

0:05:54.876,0:06:00.225
You have to select or just leave this as[br]it is, username,

0:06:00.225,0:06:06.120
if you do not enter a password, a password[br]will be generated and shown to you

0:06:06.120,0:06:07.706
and sent by e-mail.

0:06:08.435,0:06:11.445
I will now just type in the password.

0:06:11.932,0:06:15.019
It's here in clear text, for me that's fine[br]because

0:06:15.019,0:06:21.430
there's also a comment that you should[br]change the password after the installation

0:06:21.430,0:06:27.337
and I do not like to enter passwords twice[br]so you can see what you typed in

0:06:27.337,0:06:33.311
and hopefully do not make any wrong[br]mistakes.

0:06:33.636,0:06:38.311
For example, we could select the Stretch[br]distribution with backports,

0:06:38.311,0:06:42.208
so we will get a 4.15 kernel with Stretch.

0:06:43.221,0:06:47.687
There are some buttons we can say we want[br]to have some Debian developer tools.

0:06:48.246,0:06:54.145
This is what I defined in the FAI[br]configuration, so just a list of packages.

0:06:55.041,0:06:59.219
Here, you can enter you own packages.[br]

0:07:01.301,0:07:04.548
I will select the desktop.

0:07:04.832,0:07:09.340
You can have an installation without any[br]desktop, so a very small installation.

0:07:10.963,0:07:16.042
I will select the XFCE desktop, but all[br]the other desktops are here.

0:07:16.368,0:07:21.935
The language, these are just task packages[br]that are…

0:07:22.343,0:07:26.532
I think Debian has much more task packages,[br]I just searched which are

0:07:26.532,0:07:35.516
the most common languages, and what I do[br]if I say I want the spanish language,

0:07:35.516,0:07:37.986
also the keyboard layout is spanish.

0:07:39.254,0:07:43.239
I know there are different combinations[br]and with local time,

0:07:43.239,0:07:47.071
it's getting more difficult.

0:07:47.999,0:07:53.121
This installation will install the clock[br]with UTC, so if you want to set

0:07:53.121,0:07:55.315
your time, you have to do this manually.

0:07:55.751,0:07:59.951
I want to cover the most common installations.

0:08:02.389,0:08:11.372
We select english US, the desktop and,[br]as an example, the midnight commander

0:08:11.372,0:08:12.746
and GIMP.

0:08:13.242,0:08:21.050
I can add an email address so if it would[br]take longer,

0:08:21.050,0:08:26.333
for example if this service will have[br]success and a lot of people are using it,

0:08:26.333,0:08:30.992
you may wait for some minutes so your job[br]will be finished.

0:08:31.774,0:08:37.990
So here are the comments, how to reconfigure[br]the keyboard or the timezone

0:08:37.990,0:08:41.568
and then you just click "Create[br]the installation image".

0:08:42.138,0:08:49.333
Now, in the background, there's some job,[br]a script, looking "Oh, there's a new job"

0:08:49.333,0:08:54.117
and there's a summary of the configuration,[br]of the web configuration.

0:08:55.504,0:08:59.164
Down here you see these are the[br]FAI classes,

0:08:59.164,0:09:01.849
I will explain a little bit more about this.

0:09:02.092,0:09:06.552
But with this information, FAI configuration[br]is generated,

0:09:06.884,0:09:10.179
that's what normally the experienced[br]sysadmins have to create

0:09:10.909,0:09:17.541
but here you just click on some buttons[br]and it will be done for you.

0:09:21.889,0:09:26.272
In the meantime, we have some more[br]advanced features

0:09:26.272,0:09:28.744
which I will also show you later.

0:09:29.036,0:09:34.752
For example, this very simple installation[br]just creates one partition

0:09:34.752,0:09:39.923
but you can also select that you want[br]to have a separate /home partition

0:09:39.923,0:09:45.733
or using lvm just by selecting this[br]on the web interface.

0:09:47.278,0:09:55.123
You can also add your SSH public key[br]for logging as root without a password

0:09:55.123,0:10:00.649
or what's very nice, I found the new[br]Ubuntu installer does this,

0:10:00.649,0:10:05.277
you can give your github account and[br]then there's a command which

0:10:05.277,0:10:09.791
receives the public key from your[br]github account and puts it

0:10:09.791,0:10:12.959
into the root account so you can log in[br]without password.

0:10:13.239,0:10:14.778
I think that's very neat.

0:10:15.296,0:10:22.037
And if you have a repository with your own[br]packages, you could also add this and say

0:10:22.037,0:10:26.508
"Please install those packages from[br]my publicly available repository."

0:10:28.663,0:10:29.994
Let's see.

0:10:30.479,0:10:34.747
As we see, this job finished in 74 seconds.

0:10:35.891,0:10:40.004
Now, this customised installation image[br]is available for download.

0:10:40.658,0:10:43.100
You can also download the log file.

0:10:43.548,0:10:47.325
Since this is an installation image,[br]I first have to create

0:10:47.325,0:10:50.213
a partial package mirror.

0:10:51.960,0:10:56.882
This is done by the command 'fai-mirror'[br]and you can also read the log of

0:10:56.882,0:11:04.569
this call of the fai-mirror, where a list[br]of all you packages with all the dependencies

0:11:04.569,0:11:06.155
are available.

0:11:06.400,0:11:12.941
So you see, these are the list of packages[br]and later they are downloaded

0:11:12.941,0:11:21.402
and in the end, it says it created a mirror[br]of 1G of packages

0:11:21.402,0:11:24.328
and since I have a local mirror,[br]it's very fast.

0:11:24.979,0:11:31.113
This is the one part on the installation[br]image partial mirror with all the packages

0:11:31.113,0:11:36.427
and the other is that the config space[br]which you can also download.

0:11:36.973,0:11:41.353
So this is the config space that was[br]really created for you

0:11:41.353,0:11:43.627
by clicking the web interface.

0:11:44.025,0:11:47.977
If you want to do more things with FAI,[br]you can set up your own FAI server

0:11:47.977,0:11:50.047
and use this configuration space.

0:11:51.596,0:11:56.921
And, that's also very new, the two commands[br]that are used for creating this ISO image

0:11:56.921,0:11:59.562
are now also listed there.

0:12:00.010,0:12:04.723
First, create the partial mirror and then[br]create the installation image.

0:12:06.355,0:12:10.988
Ok, copy link location…

0:12:11.804,0:12:14.446
Let's see how good the network is here.

0:12:20.096,0:12:21.638
[Q] It's a rather large image.

0:12:22.167,0:12:28.789
Yeah, because it includes all the packages[br]and with Xfce, LibreOffice and so on

0:12:29.114,0:12:32.931
and the installation environment is maybe[br]about 200MB.

0:12:33.413,0:12:38.053
That's not much bigger than the Debian[br]installer that you need to download.

0:12:39.193,0:12:42.851
So, 2, 1, done.

0:12:47.731,0:12:56.278
I have a little wrapper which calls[br]a fresh kvm machine

0:12:56.278,0:13:01.115
with an empty disk and boots this ISO image

0:13:01.115,0:13:05.914
and then we will see how this installation[br]runs.

0:13:07.990,0:13:10.257
So this is dracut booting the image

0:13:11.200,0:13:17.289
and now you see there are already[br]some parted commands executed

0:13:17.289,0:13:21.955
and now the packages are installed[br]and everything runs on

0:13:21.955,0:13:24.313
and in the end some customization[br]script.

0:13:24.880,0:13:29.392
We use only shell scripts for doing[br]some customizations

0:13:29.392,0:13:33.250
and you see the files are downloaded[br]from /media/mirror

0:13:33.250,0:13:35.893
so this is local on the ISO image.

0:13:36.422,0:13:40.336
It would also be possible to create[br]an image without the packages

0:13:40.336,0:13:45.452
and then give another sources.list file[br]so the packages would be downloaded

0:13:45.452,0:13:51.585
from the internet but this default[br]in the FAI service,

0:13:51.585,0:13:54.632
we put everything onto the ISO image.

0:13:58.213,0:14:01.140
I guess it will run for 4 minutes.

0:14:03.418,0:14:07.310
[Q] ???

0:14:08.332,0:14:09.305
Yeah.

0:14:14.019,0:14:16.575
What I will show you now is…

0:14:17.019,0:14:21.083
So, this was the simple one, now I toggle[br]this web page and

0:14:21.083,0:14:25.594
you will see that there are some more[br]questions you may answer,

0:14:25.594,0:14:29.415
for example you can give a root password.

0:14:29.740,0:14:32.869
If you leave this empty, sudo will be[br]configured.

0:14:33.633,0:14:39.418
Here you can upload the SSH key or give[br]your Github account,

0:14:39.418,0:14:41.373
that would be Mrfai for me.

0:14:42.554,0:14:46.290
With the partitioning schemes, we have[br]one partition

0:14:46.290,0:14:52.508
or one partition and /home separated[br]or these two versions with LVM.

0:14:53.157,0:14:59.376
FAI itself can do much more, we could do[br]soft raid set ups, cryptsetup

0:14:59.376,0:15:03.802
but here I want to cover the most common[br]installation, so very simple

0:15:03.802,0:15:06.888
we have only 4 things that you can choose.

0:15:08.188,0:15:10.671
[Q] For encryption?

0:15:11.037,0:15:11.802
[A] Yes.

0:15:15.757,0:15:19.009
So, this was the partitioning things.

0:15:19.374,0:15:25.270
This is the new feature where you can add[br]an URL for your local package repository

0:15:25.270,0:15:30.723
and the rest is the same, you can add[br]packages you like, your email address

0:15:30.723,0:15:33.366
and then also create an installation image.

0:15:36.378,0:15:44.434
I normally set… By default, I include[br]the nonfree linux firmware.

0:15:45.408,0:15:51.824
This is because my target audience is[br]an end user and I want to make it

0:15:51.824,0:15:56.087
very comfortable for them, so yeah,[br]they can just install it

0:15:56.087,0:15:58.083
and do not have these problems.

0:15:58.411,0:16:02.431
And since this is not on an official[br]debian.org web site,

0:16:02.431,0:16:07.634
I can do this with this default.

0:16:11.373,0:16:14.020
Let's see, the installation is still running.

0:16:17.277,0:16:19.847
So, advanced features.

0:16:21.340,0:16:23.618
The next thing after this installation,

0:16:23.618,0:16:26.584
I will show you how to create cloud[br]images.

0:16:27.115,0:16:31.582
Currently, we create an installation image,

0:16:31.582,0:16:34.590
when you boot it, the installation is run[br]fully automatically.

0:16:35.160,0:16:42.761
The other type of service FAI.me gives[br]to you is that it creates a raw image

0:16:42.761,0:16:47.798
or some other formats as you see here,[br]qcow2 and whatever

0:16:47.798,0:16:51.491
which you can just boot and[br]the installation is already done.

0:16:53.857,0:16:58.330
But first, see if the installation finish.

0:17:00.066,0:17:01.538
Ah! Ok.

0:17:02.027,0:17:07.878
These are now the shell scripts that are[br]executed for the customization

0:17:07.878,0:17:13.810
of like /etc/messages of today,[br]/etc/network/interfaces is written

0:17:13.810,0:17:15.394
and so on.

0:17:15.759,0:17:21.528
You see the installation took 236s, it says[br]there are some errors

0:17:21.528,0:17:24.219
but that's not really true.

0:17:24.786,0:17:29.182
And it stops here, but we can also[br]disable this, this is only for showing

0:17:29.182,0:17:31.414
everything went well

0:17:31.414,0:17:34.628
and now we just reboot the machine.

0:17:37.351,0:17:39.259
You see the grub.

0:17:46.696,0:17:49.503
Ok, Xfce desktop.

0:17:51.049,0:17:55.597
debian was the user with password FAI.

0:18:00.025,0:18:08.474
We have "uname -a", this is 4.16,[br]the backports kernel was installed there.

0:18:08.474,0:18:13.955
We have only one partition, no LVM

0:18:13.955,0:18:21.094
and I told it to install gimp which is[br]not installed by default.

0:18:21.414,0:18:24.436
Gimp is there, so this is nice.

0:18:24.954,0:18:27.160
And the midnight commander is also there.

0:18:30.529,0:18:34.310
And now we just throw this machine.[br]Gone.

0:18:35.734,0:18:43.061
What's very nice with this wrapper script,[br]it creates the local disk

0:18:43.061,0:18:47.529
of the virtual machine in /tmp which is[br]a RAM disk and I love RAM,

0:18:47.854,0:18:50.051
it's so nice and fast.

0:18:50.983,0:18:55.578
So, this was installation image and now we[br]look at the cloud image.

0:18:56.877,0:19:01.183
First, you can say how big should your[br]disk image be.

0:19:04.120,0:19:10.328
Here, I say 8GB, you will see it's not[br]an 8GB image

0:19:10.328,0:19:12.810
that you have to download later.

0:19:13.218,0:19:17.651
By default, I use zstd compression.

0:19:18.424,0:19:21.514
Anyone who does not know[br]this compression?

0:19:22.495,0:19:27.543
This is very fast, very new, created[br]by Facebook if I'm correct.

0:19:29.159,0:19:37.583
It's for very big files and what you should[br]never use is gzip with sparse images.

0:19:38.394,0:19:41.643
The disk image is sparse and gzip[br]cannot handle this

0:19:41.643,0:19:45.710
so if you compress it and uncompress it[br]it will be very large

0:19:45.710,0:19:52.579
and all the other, xz, zstd, can handle[br]sparse files very nicely.

0:19:54.000,0:20:02.125
So, the hostname is set, the root password,[br]username with a password.

0:20:02.524,0:20:04.526
Now we want to install Buster.

0:20:07.087,0:20:08.957
Maybe with no…

0:20:12.007,0:20:15.385
Oh, we also do the Xfce desktop.

0:20:16.443,0:20:20.097
Any packages you'd like to have in this[br]cloud image.

0:20:21.108,0:20:24.687
"desktop" and "cloud" image does not[br]make that much sense, maybe.

0:20:26.717,0:20:30.780
Emacs25, ok.

0:20:32.888,0:20:36.679
And now "Create disk image".

0:20:37.451,0:20:42.127
This will take a little bit longer because[br]we are doing the installation

0:20:42.127,0:20:45.417
inside a file image.

0:20:46.350,0:20:52.252
But no problem, I can tell you what[br]other ideas I have.

0:20:52.823,0:21:01.591
So, currently we have the installation and[br]the cloud or virtual machine images for amd64

0:21:01.591,0:21:08.462
FAI itself can also do cross-architecture[br]images so it would be some work

0:21:08.462,0:21:14.191
to extend the web page to say[br]"Please create an arm64 image"

0:21:14.191,0:21:21.998
It would be very nice to have predefined[br]configs for raspberry pi or

0:21:21.998,0:21:24.110
all the very different boards

0:21:24.556,0:21:26.785
but that would also be possible.

0:21:28.577,0:21:33.041
I guess the next thing I will implement[br]is other distributions

0:21:33.041,0:21:35.852
because I know people are always asking it.

0:21:36.299,0:21:38.414
Not you but the Ubuntu guys.

0:21:39.798,0:21:44.925
Yesterday I did the first test with Ubuntu[br]bionic, the LTS release

0:21:44.925,0:21:47.325
and FAI just works out of the box with it.

0:21:47.854,0:21:53.867
So what I have to do is to integrate it[br]in these FAI.me processing scripts.

0:21:55.405,0:22:00.323
Ready-to-go cloud images for the big[br]cloud providers.

0:22:02.643,0:22:06.438
That's only a different FAI config space[br]that I have to use.

0:22:06.884,0:22:11.070
Currently, for example, in what I call[br]cloud images, I do not install

0:22:11.070,0:22:12.693
the package cloud image.

0:22:13.912,0:22:16.674
That's needed for all the ones.

0:22:17.842,0:22:24.865
I'm also working in the Debian cloud team[br]and this team decided 2 years ago

0:22:24.865,0:22:29.873
that the tool chain in the future for[br]the official Debian package will be FAI.

0:22:31.052,0:22:36.937
Amazon is already using it, so if you[br]boot or if you use a Debian cloud image

0:22:36.937,0:22:41.284
in Amazon, Noah Meyerhans did this and[br]he's using the FAI tool chain for it.

0:22:41.975,0:22:48.562
Google is not yet using it because there was[br]a very small problem

0:22:48.562,0:22:51.737
in a config file we had one space too much

0:22:51.737,0:22:55.273
which caused grub to hang forever

0:22:55.273,0:23:00.657
and that was the reason why they decided[br]for Stretch to use their own tool chain.

0:23:01.041,0:23:06.125
But the things are working so we have[br]the config space also for Google.

0:23:07.293,0:23:12.264
And also for Azure, some people from[br]Credativ did this.

0:23:14.911,0:23:18.482
The Debian cloud team already has[br]the FAI configuration for

0:23:18.482,0:23:23.521
the big tool providers,[br]cloud providers.

0:23:24.821,0:23:30.955
We could also think on a more generic[br]FAI installation image.

0:23:31.889,0:23:37.337
It's an image that you would boot up[br]and then enter your job id of the web page

0:23:37.337,0:23:39.778
and then the configuration would be[br]downloaded

0:23:39.778,0:23:42.945
and the packages would be just[br]received from the internet.

0:23:43.637,0:23:45.144
That was one…

0:23:45.511,0:23:49.697
So, the image would be much smaller[br]because the packages do not need to be

0:23:49.697,0:23:52.865
on the installation image.

0:23:54.690,0:23:57.859
It's also possible to create live images[br]with FAI.

0:23:58.590,0:24:00.136
It is a little bit more…

0:24:01.272,0:24:06.558
Currently, you need some manual work[br]but that should be also possible

0:24:06.558,0:24:11.157
to use FAI for creating live image[br]and then also to provide this

0:24:11.157,0:24:13.802
on the FAI.me web service.

0:24:14.574,0:24:17.738
If you want to customize much more[br]inside the image, you just say

0:24:17.738,0:24:21.882
"Oh, I have some Ansible scripts that[br]I want to execute at the very end"

0:24:22.070,0:24:26.967
then I say "Ok, this is just a starting[br]point, use the FAI.me service

0:24:26.967,0:24:31.844
and if you're happy with the FAI tools,[br]then set up your own FAI server,

0:24:31.844,0:24:37.856
create your own configuration space[br]and then you can do all the crazy things."

0:24:39.773,0:24:42.335
So, how does FAI.me work internally.

0:24:43.148,0:24:47.861
We have a web server where there are[br]some CGI scripts and

0:24:47.861,0:24:53.548
this is not the build server, so on the[br]web server, you click "Submit"

0:24:53.548,0:25:02.054
"Create my image", all the input[br]is validated so you cannot make nasty things

0:25:02.054,0:25:08.682
and then the CGI writes or creates[br]a subdirectory and puts 2 files in it,

0:25:08.916,0:25:10.301
a config and a meta file

0:25:11.561,0:25:14.937
and writes a status[br]"waiting for processing".

0:25:15.344,0:25:19.451
Then, the other server, the build server[br]reads this config and

0:25:19.451,0:25:25.057
this is just an NFS mounted directory,[br]and sees

0:25:25.057,0:25:27.169
"Oh there's a new job I have to process".

0:25:30.147,0:25:33.615
In this processing script we pass for[br]some errors.

0:25:34.304,0:25:39.624
What's happening very often that people[br]type in a package that's not available

0:25:39.624,0:25:45.239
and this will be detected and then a new[br]version of the web page will pop up and say

0:25:45.239,0:25:48.692
"Oh, when creating the package mirror,[br]there was an error

0:25:48.692,0:25:50.929
because this package was not known."

0:25:52.882,0:25:54.062
Sometimes I have to…

0:25:54.508,0:25:57.645
Every night, I create new nfsroots[br]for Buster.

0:25:59.105,0:26:03.172
If there are security updates, I have[br]to create new nfsroots

0:26:03.172,0:26:05.073
for Stretch and backports.

0:26:05.598,0:26:10.079
I have some cleanup, so if a lot of jobs[br]are created,

0:26:10.079,0:26:12.758
the images are on the disk after,

0:26:12.758,0:26:17.270
normally I say after one day I just[br]remove the images

0:26:17.270,0:26:20.567
so you have one day to download[br]the images.

0:26:22.761,0:26:25.763
There's 3 different configurations

0:26:25.763,0:26:28.894
/etc/fai-stretch, /etc/buster,[br]/etc/fai-stretch-bpo (backports)

0:26:29.129,0:26:30.846
We need for the installation image

0:26:30.846,0:26:36.336
We need a different nfsroot, but[br]the config space that is shared

0:26:36.336,0:26:38.283
about all configurations,

0:26:38.283,0:26:41.209
so it doesn't matter if I install

0:26:41.209,0:26:46.293
Stretch or Stretch backports or[br]Buster,

0:26:46.293,0:26:49.583
I can use the same FAI configuration.

0:26:49.949,0:26:53.520
Also, for building the cloud images,[br]I use the same FAI configuration.

0:26:56.327,0:27:04.656
A new job is detected, then a copy of[br]the configuration space will be made

0:27:04.656,0:27:07.913
and it will be customized a little bit.

0:27:08.352,0:27:10.430
So there are a very very few changes,

0:27:10.430,0:27:16.686
for example I have to put the SSH key[br]into your customized configuration space

0:27:16.686,0:27:20.626
or the list of packages or the user and[br]root password.

0:27:21.475,0:27:25.697
Then we have two things, if we want to[br]create the installation image,

0:27:25.697,0:27:31.148
I first have to create the partial package[br]mirror and then create the installation image

0:27:31.148,0:27:35.700
For the cloud images, we do not need[br]the nfsroot, we just need

0:27:35.700,0:27:38.914
the configuration space which is[br]customized a little bit

0:27:38.914,0:27:41.543
and then we can just create the disk image

0:27:41.543,0:27:46.590
so there's one step less compared to[br]creating the installation ISO.

0:27:47.765,0:27:51.507
The status on the web page will be[br]updated, log files written

0:27:51.507,0:27:55.375
and if the user said "Please send me[br]an email if my job is ready",

0:27:55.375,0:27:57.736
this will also be sent to the user.

0:27:58.593,0:28:03.430
Then we have the ISO or the disk image[br]and this will be copied back

0:28:03.430,0:28:07.659
to the web server where the user can then[br]download it.

0:28:08.706,0:28:12.084
And since I have a lot of RAM[br]in this machine,

0:28:12.084,0:28:15.533
everything is run in RAM, very very nice.

0:28:17.563,0:28:22.804
As I said, we need an nfsroot,[br]a configuration space and FAI classes.

0:28:23.171,0:28:26.260
This is a very central component in FAI

0:28:26.625,0:28:28.819
and this is just a list of names.

0:28:29.387,0:28:36.010
So in HOME_LVM, this is the class name,[br]the FAI class we describe

0:28:36.010,0:28:38.619
and I think this is that example:

0:28:38.619,0:28:43.452
HOME_LVM describes how to partition[br]the local hard disk.

0:28:44.795,0:28:50.657
This is our very flexible tool where[br]we can do LVM, cryptsetups,

0:28:50.657,0:28:52.807
software RAIDs and so on.

0:28:53.417,0:28:57.840
But for the FAI.me service, I just created[br]4 different types of partitioning

0:28:57.840,0:29:00.443
and this is the HOME_LVM example.

0:29:03.733,0:29:07.469
So we have a list of classes and,[br]as I said,

0:29:07.469,0:29:12.183
just two commands for the installation[br]image with a list of classes

0:29:12.183,0:29:17.762
and for the cloud image, I have to say[br]how big should the disk image be,

0:29:17.762,0:29:21.994
the list of classes and what's the target[br]file that should be created.

0:29:23.856,0:29:25.891
Let's see if this is ready.

0:29:28.083,0:29:29.137
Yes, it's ready.

0:29:29.504,0:29:30.357
So…

0:29:31.820,0:29:33.567
It's 1.1GB.

0:29:34.620,0:29:36.898
Is this really the… oh yeah, raw.

0:29:41.076,0:29:45.555
No problem, let's download it, it should[br]be fast.

0:29:48.911,0:29:53.522
This is the normal architecture if you use[br]FAI in a client/server set up.

0:29:54.092,0:29:57.507
You should just look on the left side[br]where you see

0:29:57.507,0:30:01.535
you need the config space, an nfsroot[br]and a mirror

0:30:01.535,0:30:04.048
and these parts will put onto the CD.

0:30:05.309,0:30:08.683
If you set up a network installation thing,

0:30:08.914,0:30:13.139
this is how things get from the server[br]to the client.

0:30:15.506,0:30:21.403
For the software installation, we have[br]another subdirectory called package_config

0:30:21.403,0:30:27.014
and there you also see several files[br]where the file name is a FAI class.

0:30:27.542,0:30:34.003
Since in the FAI.me service every client[br]belongs to the class DEBIAN,

0:30:34.003,0:30:38.723
it will install the packages that are listed[br]on the top

0:30:39.768,0:30:42.774
and here we have an other class, NONFREE

0:30:42.774,0:30:46.679
These packages are only installed if[br]you also said

0:30:46.679,0:30:49.685
"Please install the nonfree packages"

0:30:49.685,0:30:53.018
and this is mapped to a FAI class[br]called NONFREE.

0:30:53.832,0:30:56.799
And there's an other class for AMD64[br]and so on.

0:30:58.870,0:31:01.516
Some references.

0:31:02.127,0:31:04.771
In the past, it looked more like this[br]when I said

0:31:04.771,0:31:12.899
"Oh, who's using FAI?" and during[br]the last month I collected some logos

0:31:12.899,0:31:15.537
just because it's much nicer.

0:31:18.865,0:31:21.828
Let's see if the download was ready.

0:31:22.519,0:31:34.922
We unzstd the FAI.me image,[br]faime-013Z image

0:31:41.253,0:31:46.135
On the web site, I said I want to have[br]a 8GB partition,

0:31:46.951,0:31:51.213
so now let's see how big it is.

0:31:51.621,0:31:58.692
The file is 8, but since it's a sparse file[br]it's only 3.5GB

0:31:58.692,0:32:03.324
and the compressed was 1.1GB.

0:32:04.381,0:32:08.879
Now I use my wrapper

0:32:11.169,0:32:17.374
and I say "Boot from disk" and this is[br]the FAI.me raw image, disk image

0:32:18.676,0:32:20.630
that should be booted up.

0:32:33.558,0:32:34.697
That's it.

0:32:35.836,0:32:37.419
debian/fai

0:32:53.367,0:32:57.176
Let's see if emacs is installed, yes.

0:32:58.886,0:33:04.621
Gimp is already there, hopefully, and[br]the blue midnight commander.

0:33:14.131,0:33:15.923
Let's see.

0:33:21.447,0:33:23.113
Questions.

0:33:32.188,0:33:37.431
[Q] I'm using the preseed file for[br]the debian-installer,

0:33:37.431,0:33:40.842
do you have a conversion between[br]your syntax and your configuration files

0:33:40.842,0:33:45.556
and the preseed file or maybe can you add[br]a download button for the preseed file

0:33:45.556,0:33:49.010
to your web site because I think it's[br]rather nice to have it displayed

0:33:49.010,0:33:50.637
in web site first.

0:33:51.045,0:33:53.109
[A] I'm not using the debian-installer.

0:33:54.044,0:33:58.931
I use preseeding, yes, the debconf[br]preseeding for the normal packages

0:33:58.931,0:34:02.571
you can do this also in FAI and it's[br]the same format

0:34:02.571,0:34:06.396
you get with debconf-get-selections.

0:34:07.371,0:34:11.795
And what you get is you can download[br]your own FAI config space

0:34:11.795,0:34:14.881
and this includes all information you need[br]to set up,

0:34:14.881,0:34:20.614
to do this mirror FAI CD or the FAI disk[br]image command.

0:34:21.025,0:34:27.814
But you cannot convert this config into[br]a d-i preseeding or vice versa,

0:34:27.814,0:34:30.008
that's not possible.

0:34:31.268,0:34:37.634
Because for example, for the partitioning[br]part I do not like to create

0:34:37.634,0:34:42.636
from my disk config partman preseeding[br]file.

0:34:43.817,0:34:46.338
You can pay me a lot of money, I will[br]never do this.

0:34:46.705,0:34:51.948
You know that the partman preseeding[br]is very ugly and very heavy.

0:34:53.721,0:35:01.159
For other things, yes, selection of,[br]for example, the selection of the language

0:35:01.159,0:35:04.570
These are the normal preseeding we use.

0:35:05.026,0:35:07.749
And the list of packages, task selec…

0:35:08.074,0:35:13.840
I think it's much easier to do this in[br]the FAI configuration than to create

0:35:13.840,0:35:17.710
a debian-installer preseeding.

0:35:19.370,0:35:22.828
And why use d-i if this works for you?

0:35:23.112,0:35:24.729
[Q] d-i works as well for me.

0:35:25.095,0:35:27.339
[A] Yes, then fine, use it.

0:35:29.905,0:35:36.008
[Q] Hi Thomas. Thank you very much for[br]this new feature in the FAI project,

0:35:36.008,0:35:42.998
it's very nice and I found very great that[br]you have the output of the commands

0:35:42.998,0:35:49.019
that you used to create the ISO image or[br]the cloud file.

0:35:50.392,0:35:59.641
A question that I have is, in which servers[br]are located the files that we create,

0:35:59.641,0:36:01.307
the ISO or the cloud.

0:36:01.632,0:36:04.635
Is it a server that you own host or…

0:36:04.635,0:36:05.860
[A] Trust me.

0:36:09.160,0:36:10.500
Currently,

0:36:10.500,0:36:18.832
both the web server and the FAI.me[br]processing build server are run

0:36:18.832,0:36:24.960
on two machines at the university where[br]I work as a system administrator

0:36:24.960,0:36:27.568
so that's also where we have a very fast[br]connection.

0:36:30.206,0:36:35.859
The CGI script and shell script that is[br]processing these jobs is currently

0:36:35.859,0:36:37.528
not open source.

0:36:38.180,0:36:40.522
There are plans to do this, I'm not sure[br]when.

0:36:43.336,0:36:48.012
If you want to reproduce the things, you[br]have the config file and you can download

0:36:48.012,0:36:52.245
the FAI software and use these one or two[br]commands to reproduce it.

0:36:53.180,0:36:57.203
Some people said "Oh, very nice service,[br]I would like to set up in my company".

0:36:57.775,0:37:00.499
Then please yes, contact me and…

0:37:01.149,0:37:06.677
Currently there are no concrete plans[br]to make these background scripts open source

0:37:06.677,0:37:09.196
but it will be in some future.

0:37:09.764,0:37:16.628
But currently, you have to trust me as[br]you also have to trust the package maintainers

0:37:16.628,0:37:18.659
that will be installed there.

0:37:19.018,0:37:23.375
But you can verify it or say "I do not[br]trust Thomas but I will just grab

0:37:23.375,0:37:26.588
the FAI config space and this on my own".

0:37:27.156,0:37:28.297
[Q] Thank you.

0:37:30.333,0:37:33.095
[Q] There's a question from the internet.

0:37:34.641,0:37:39.597
Why not use a proper job queuing system[br]like grid engine or similar?

0:37:40.816,0:37:45.696
[A] I'm using grid engine at work for[br]different things.

0:37:47.652,0:37:53.824
It started as a very simple project, so[br]in the end it's just a loop which

0:37:53.824,0:37:56.885
checks if there's new jobs on that.

0:37:57.169,0:38:01.719
Currently, I do not process jobs[br]in parallel, currently there's no need for it

0:38:02.898,0:38:07.245
If this project will be very successful,[br]yeah, I have to use a queuing system.

0:38:07.854,0:38:10.492
It's, yeah, a very simple script.

0:38:10.775,0:38:14.677
But it would be also possible with[br]a proper queuing system.

0:38:17.116,0:38:18.254
More questions?

0:38:18.781,0:38:20.491
[Q] I have a bunch of questions.

0:38:21.913,0:38:25.113
First, what is it that you use for[br]partitioning?

0:38:26.825,0:38:31.558
[A] I'm using a Perl script that we wrote[br]several years ago in FAI

0:38:31.558,0:38:37.709
and we defined this config file,[br]this package config

0:38:37.709,0:38:46.083
and the Perl script parses this script[br]and then executes the parted and mkfs command

0:38:46.083,0:38:51.447
which you can see in the log files, so if[br]you want to see what does FAI do

0:38:51.447,0:38:55.673
after parsing this, which commands are[br]executed, you see everything

0:38:55.673,0:38:57.216
on the log files.

0:38:57.641,0:39:03.334
[Q] Right, but so you turn this text[br]into partitioning…

0:39:04.190,0:39:05.328
[A] commands, yeah.

0:39:05.573,0:39:09.351
[Q] But the text looks like this,[br]like with the spaces and everything.

0:39:09.351,0:39:14.345
[A] You can use more or less spaces[br]or do you like,

0:39:14.345,0:39:16.509
should I convert it to XML?

0:39:18.641,0:39:20.537
[Q] Ok, then my next question is

0:39:20.537,0:39:25.090
what are you using the nfsroot for when[br]you're generating the images?

0:39:25.983,0:39:32.079
[A] The nfsroot is used only for the[br]installation image.

0:39:32.079,0:39:37.090
When I do the installation, I need to boot[br]the machine as a diskless client,

0:39:37.090,0:39:41.774
so it's just what the debian-installer[br]loads into RAM,

0:39:41.774,0:39:43.358
you need a running Linux system.

0:39:43.508,0:39:46.196
This is our nfsroot, on the installation[br]image.

0:39:47.536,0:39:52.081
When you boot installation image, this[br]nfsroot with all the commands we need

0:39:52.081,0:39:55.706
are started without using the local disk[br]and then we can do

0:39:55.706,0:39:59.035
everything on the disk, /root and /target[br]and so on.

0:39:59.606,0:40:04.519
The nfsroot is the system that is running[br]during the installation.

0:40:07.361,0:40:11.590
[Q] Ok, but there's no need for this to be[br]nfs, it could be a

0:40:11.590,0:40:14.397
[A] It's called nfsroot.

0:40:14.397,0:40:19.794
This is very common that people call it[br]nfsroot

0:40:19.794,0:40:23.862
and if you have this network installation[br]thing, it's really an nfsroot.

0:40:24.145,0:40:26.256
But you're right.

0:40:26.256,0:40:32.186
On the installation ISO, it's not nfs,[br]it's just a local file system, yes.

0:40:35.229,0:40:38.891
[Q] Alright. So, I think it's my last[br]comment.

0:40:38.891,0:40:45.355
You have the ISO from which you install[br]and when you install from the ISO

0:40:45.355,0:40:50.669
you're installing then packages on the[br]machine, and then you have the image

0:40:50.669,0:40:54.660
which is like a disk image that has[br]the packages already installed,

0:40:54.660,0:40:57.585
so you skip the installing step.

0:40:59.089,0:41:02.753
Have you thought about having[br]an intermediate thing

0:41:02.753,0:41:06.810
where you download an image that[br]already has the packages installed?

0:41:09.987,0:41:11.733
[A] That's also possible.

0:41:11.733,0:41:18.199
When you do an installation, before you[br]can change root in the new system

0:41:18.199,0:41:21.650
for adding packages, you have to call[br]debootstrap.

0:41:22.501,0:41:26.363
What we do, we call debootstrap once and[br]create a tar file out of it.

0:41:27.415,0:41:31.935
This is our minimal… in the former days[br]it was the floppy disk,

0:41:31.935,0:41:34.491
our base tar.gz file,

0:41:34.491,0:41:39.779
so you could exchange the minimal tar file[br]with whatever tar file you have.

0:41:40.756,0:41:44.169
That's for example what we do if we install[br]Ubuntu.

0:41:44.864,0:41:49.739
We boot the installation system which is[br]a Debian system

0:41:49.739,0:41:57.096
and then create the local filesystem and[br]extract an Ubuntu base image

0:41:57.096,0:42:02.012
and then we can change root into[br]the Ubuntu or the same for CentOS and so on

0:42:02.012,0:42:07.014
then we can change root into the other[br]Linux system and add packages there.

0:42:07.540,0:42:13.108
If you have already a bigger image with[br]some more packages added there,

0:42:13.108,0:42:18.188
it's very easy to say "Do not extract the[br]Debian Stretch image

0:42:18.188,0:42:22.494
but use my image which also[br]includes other tools."

0:42:23.273,0:42:27.248
And if you are fine with that, you can[br]just extract the tar file.

0:42:30.583,0:42:33.097
[Q] Ok. Any more questions?

0:42:41.268,0:42:43.136
[Q] The heading is in german.

0:42:43.623,0:42:44.352
[A] What?

0:42:44.679,0:42:45.568
[Q] The heading is in german.

0:42:45.568,0:42:48.943
[A] Oh, because it's a copy of my german[br]slides.

0:42:50.641,0:42:53.050
Thank you for this.

0:42:53.326,0:42:55.287
And, what's also missing.

0:42:55.287,0:43:00.357
The web page, where you can select german[br]or other languages,

0:43:00.357,0:43:04.543
it would be nice if people are interested[br]to help translate them

0:43:04.543,0:43:09.792
so that it's more easy for people that[br]do not speak english

0:43:09.792,0:43:14.871
to use the website and create their own[br]installation image with their language.

0:43:23.500,0:43:28.099
[Q] Someone on the stream said that[br]the fai.me web site is not yours

0:43:28.099,0:43:33.343
and it's a hack thing, it's a scam, you go[br]there and get hacked.

0:43:33.790,0:43:36.755
Do you have any plans to try to buy[br]the domain because it's pretty confusing.

0:43:37.159,0:43:40.732
The first thing I would have done[br]by seeing that talk would have been

0:43:40.732,0:43:41.991
to go to fai.me.

0:43:42.436,0:43:48.812
[A] Yeah, I was thinking about which name[br]I should choose.

0:43:49.339,0:43:54.222
I didn't check which web domains are free[br]and in then end I thought

0:43:54.222,0:43:58.612
"Do I like to have a different domain name[br]for the service?"

0:43:58.612,0:44:03.006
But since it's only a part of the FAI[br]project, I thought,

0:44:03.006,0:44:07.771
and technically it was also easier just[br]to host it under a subdirectory

0:44:07.771,0:44:16.921
and yeah, if people now grab the fai.me[br]domain and do other things with it, yeah.

0:44:17.445,0:44:24.900
I think one question would be to use[br]a debian.net or maybe debian.org domain

0:44:24.900,0:44:27.735
because people trust much more.

0:44:28.024,0:44:32.374
I get a lot of comments "Oh this would[br]be very nice if this would be hosted

0:44:32.374,0:44:34.283
on a Debian machine"

0:44:34.607,0:44:38.510
but this would be much more complicated[br]because the DSA team

0:44:38.510,0:44:43.064
has much more restriction what to execute[br]on their machines.

0:44:43.796,0:44:48.631
currently, we need root access because[br]we mount some things

0:44:48.631,0:44:55.019
and DSA would not give me root access[br]on any Debian machine.

0:44:55.831,0:44:59.535
That's the same problem we have[br]in the Debian cloud team

0:44:59.535,0:45:04.367
where we want to create the official[br]images for the cloud providers

0:45:04.367,0:45:08.560
where the Debian cloud team will also[br]not have root access

0:45:08.560,0:45:13.388
and so there's much more work to get[br]empty virtual machines

0:45:13.388,0:45:18.155
starting up, putting data into it,[br]creating the images,

0:45:18.155,0:45:20.914
receiving them from inside the image.

0:45:22.173,0:45:26.846
And since on those machines I have[br]root access, that's much easier for me.

0:45:29.966,0:45:33.393
We are out of time, so thank you Thomas.

0:45:34.652,0:45:40.099
[Applause]