Today, I talk about FAI.me, which is a build for images.
First, anybody that never heard anything about FAI?
Ok
I started this project in 1999.
I'm not sure…
No, I'm sure that during those times, the Debian installer did not have
the preseeding stuff, so we needed something automatically.
I installed the first cluster with FAI and I always do talks on FAI or
today in the lightning talks, I talk a little bit about dracut,
which is used in FAI.
So, what was the motivation.
A neighbour of mine, she came to me with
"My Windows desktop is broken, can you reinstall it?"
And in the end, I installed her Linux, and I was shortly thinking about
"Should I use FAI for installing her desktop with Linux?"
And in the end, I did not use it because FAI is too complicated,
like the Debian installer, I guess it's not really that easy for beginners
because there are a lot of questions
but also FAI is not really for beginners.
So this was the motivation about thinking about FAI.
The target group was always advanced sysadmins
but I thought maybe it's possible to make FAI usable also for people
that are not that advanced sysadmins.
The idea is that an installer should cover most installations.
The Debian installer is really perfect because I think it covers
all different kinds and strange environments
You can do a lot of things, you can configure very strange combination of language,
keyboard layout and so on
but I was thinking about an installer that covers 90 or 95% of the installations
A lot of special cases can be ignored and since the Debian installer has like
more than 20 questions, I thought it would be much nicer if there were only
3 to 5 questions and I looked at Linux Mint and Mageia installers, CentOS installer,
and they all ask much less questions.
In the Debian installer, we sometimes have also things that are asked
during the installation, so not everything is asked at the very beginning.
For example, the task selection, where you select your desktop,
is done after the base installation.
This was also very important, I would like to have something that
asks everything at the very beginning.
Then, maybe some tool could create a customized installation image
and this installation image should run then completely unattended
so you can get yourself a coffee and when you come back, your machine is ready.
There are 3 things to customize installation image,
you just put this image, you do not have to touch anything, and then it's ready.
I thought "Oh yes, this is FAI, maybe FAI can do this."
As I said, FAI is only, or was until now only a tool for experienced sysadmins
and you have to adjust several config files, these are ASCII files
but still you have to touch 5 to 10 config files to make a customization.
So, how can I make FAI usable for beginners?
That's the beginning of FAI.me.
There's a web page, we'll show it in more detail later,
where you can just click some things, and then you get a customized image.
This image can be put onto a CD, DVD or USB stick, just with dd
and the customization is just by using the web interface
so there's no need for you to edit a text file, a config file inside FAI.
I hope I covered most important thangs that you want to adjust
or a little bit customize.
You can add additional packages, I think that's the most important thing
that people say "I want to have the normal Debian installation
but with some additional packages."
And you can select different different distributions, so it's not only
the installation image for the stable release, you can create
3 variants of the installation.
This is the web page and thanks to Juri, he did a great job
during the first and second day, he added a new feature that we now have
a toggle button.
Is it big enough or should I zoom in?
Ok.
So, we have a toggle button, what you see now is just the bare minimum or questions
and we can toggle it to more advanced settings.
You have to select or just leave this as it is, username,
if you do not enter a password, a password will be generated and shown to you
and sent by e-mail.
I will now just type in the password.
It's here in clear text, for me that's fine because
there's also a comment that you should change the password after the installation
and I do not like to enter passwords twice so you can see what you typed in
and hopefully do not make any wrong mistakes.
For example, we could select the Stretch distribution with backports,
so we will get a 4.15 kernel with Stretch.
There are some buttons we can say we want to have some Debian developer tools.
This is what I defined in the FAI configuration, so just a list of packages.
Here, you can enter you own packages.
I will select the desktop.
You can have an installation without any desktop, so a very small installation.
I will select the XFCE desktop, but all the other desktops are here.
The language, these are just task packages that are…
I think Debian has much more task packages, I just searched which are
the most common languages, and what I do if I say I want the spanish language,
also the keyboard layout is spanish.
I know there are different combinations and with local time,
it's getting more difficult.
This installation will install the clock with UTC, so if you want to set
your time, you have to do this manually.
I want to cover the most common installations.
We select english US, the desktop and, as an example, the midnight commander
and GIMP.
I can add an email address so if it would take longer,
for example if this service will have success and a lot of people are using it,
you may wait for some minutes so your job will be finished.
So here are the comments, how to reconfigure the keyboard or the timezone
and then you just click "Create the installation image".
Now, in the background, there's some job, a script, looking "Oh, there's a new job"
and there's a summary of the configuration, of the web configuration.
Down here you see these are the FAI classes,
I will explain a little bit more about this.
But with this information, FAI configuration is generated,
that's what normally the experienced sysadmins have to create
but here you just click on some buttons and it will be done for you.
In the meantime, we have some more advanced features
which I will also show you later.
For example, this very simple installation just creates one partition
but you can also select that you want to have a separate /home partition
or using lvm just by selecting this on the web interface.
You can also add your SSH public key for logging as root without a password
or what's very nice, I found the new Ubuntu installer does this,
you can give your github account and then there's a comment which
receives the public key from your github account and puts it
into the root account so you can log in without password.
I think that's very neat.
And if you have a repository with your own packages, you could also add this and say
"Please install those packages from my publicly available repository."
Let's see.
As we see, this job finished in 74 seconds.
Now, this customised installation image is available for download.
You can also download the log file.
Since this is an installation image, I first have to create
a partial package mirror.
This is done by the command 'fai-mirror' and you can also read the log of
this call of the fai-mirror, where a list of all you packages with all the dependencies
are available.
So you see, these are the list of packages and later they are downloaded
and in the end, it says it created a mirror of 1G of packages
and since I have a local mirror, it's very fast.
This is the one part on the installation image partial mirror with all the packages
and the other is that the config space which you can also download.
So this is the config space that was really created for you
by clicking the web interface.
If you want to do more things with FAI, you can set up your own FAI server
and use this configuration space.
And, that's also very new, the two commands that are used for creating this ISO image
are now also listed there.
First, create the partial mirror and then create the installation image.
Ok, copy link location…
Let's see how good the network is here.
[Q] It's a rather large image.
Yeah, because it includes all the packages and with Xfce, LibreOffice and so on
and the installation environment is maybe about 200MB.
That's not much bigger than the Debian installer that you need to download.
So, 2, 1, done.
I have a little wrapper which calls a fresh kvm machine
with an empty disk and boots this ISO image
and then we will see how this installation runs.
So this is dracut booting the image
and now you see there are already some parted commands executed
and now the packages are installed and everything runs on
and in the end some customization script.
We use only shell scripts for doing some customizations
and you see the files are downloaded from /media/mirror
so this is local on the ISO image.
It would also be possible to create an image without the packages
and then give another sources.list file so the packages would be downloaded
from the internet but this default in the FAI service,
we put everything onto the ISO image.
I guess it will run for 4 minutes.
[Q] ???
Yeah.
What I will show you now is…
So, this was the simple one, now I toggle this web page and
you will see that there are some more questions you may answer,
for example you can give a root password.
If you leave this empty, sudo will be configured.
Here you can upload the SSH key or give you Github account,
that would be Mrfai for me.
With the partitioning schemes, we have one partition
or one partition and /home separated or these two versions with LVM.
FAI itself can do much more, we could do soft raid set ups, cryptsetup
but here I want to cover the most common installation, so very simple
we have only 4 things that you can choose.
[Q] For encryption?
[A] Yes.
So, this was the partitioning things.
This is the new feature where you can add an URL for your local package repository
and the rest is the same, you can add packages you like, your email address
and then also create an installation image.
I normally set by default, I include the nonfree linux firmware.
This is because my target audience is an end user and I want to make it
very comfortable for them, so yeah, they can just install it
and do not have these problems.
And since this is not on an officiel debian.org web site,
I can do this with this default.
Let's see, the installation is still running.
So, advanced features.
The next thing after this installation,
I will show you how to create cloud images.
Currently, we create an installation image,
when you boot it, the installation is run fully automatically.
The other type of service FAI.me gives to you is that it creates a raw image
or some other formats as you see here, qcow2 and whatever
which you can just boot and the installation is already done.
But first, see if the installation finish.
Ah! Ok.
These are now the shell scripts that are executed for the customization
of like /etc/messages of today, /etc/network/interfaces is written
and so on.
You see the installation took 236s, it says there are some errors
but that's not really true.
And it stops here, but we can also disable this, this is only for showing
everything went well
and now we just reboot the machine.
You see the grub.
Ok, Xfce desktop.
debian was a user with password FAI.
We have "uname -a", this is 4.16, the backports kernel was installed there.
We have only one partition, no LVM
and I told it to install gimp which is not installed by default.
Gimp is there, so this is nice.
And the midnight commander is also there.
And now we just throw this machine. Gone.
What's very nice with this wrapper script, it creates the local disk
of the virtual machine in /tmp which is a RAM disk and I love RAM,
it's so nice and fast.
So, this was installation image and now we look at the cloud image.
First, you can say how big should your disk image be.
Here, I say 8GB, you will see it's not an 8GB image
that you have to download later.
By default, I use zstd compression.
Anyone who dose not know this compression?
This is very fast, very new, created by Facebook if I'm correct.
It's for very big files and what you should never use is gzip with sparse images.
The disk image is sparse and gzip cannot handle this
so if you compress it and uncompress it it will be very large
and all the other, xz, zstd, can handle sparse files very nicely.
So, the hostname is set, the root password, username with a password.
Now we want to install Buster.
Maybe with no…
Oh, we also do the Xfce desktop.
Any packages you'd like to have in this cloud image.
"desktop" and "cloud" image does not make that much sense, maybe.
Emacs25, ok.
And now "Create disk image".
This will take a little bit longer because we are doing the installation
inside a file image.
But no problem, I can tell you what other ideas I have.
So, currently we have the installation and the cloud or virtual machine images for amd64
FAI itself can also do cross-architecture images so it would be some work
to extend the web page to say "Please create an arm64 image"
It would be very nice to have predefined configs for raspberry pi or
all the very different boards
but that would also be possible.
I guess the next thing I will implement is other distributions
because I know people are always asking it.
Not you but the Ubuntu guys.
Yesterday I did the first test with Ubuntu bionic, the LTS release
and FAI just works out of the box with it.
So what I have to do is to integrate it in these FAI.me processing scripts.
Ready-to-go cloud images for the big cloud providers.
That's only a different FAI config space that I have to use.
Currently, for example, in what I call cloud images, I do not install
the package cloud image.
That's needed for all the ones.
I'm also working in the Debian cloud team and this team decided 2 years ago
that the tool chain in the future for the official Debian package will be FAI.
Amazon is already using it, so if you boot or if you use a Debian cloud image
in Amazon, Noah Meyerhans did this and he's using the FAI tool chain for it.
Google is not yet using because there was a very small problem
in a config file we had one space too much
which caused grub to hang forever
and that was the reason why they decided for Stretch to use their auto tool chain.
But the things are working so we have the config space also for Google.
And also for Azure, some people from Credativ did this.
The Debian cloud team already has the FAI configuration for
the big tool providers, cloud providers.
We could also think on a more generic FAI installation image.
It's an image that you would boot up and then enter your job id of the web page
and then the configuration would be downloaded
and the packages would be just received from the internet.
That was one…
So, the image would be much smaller because the packages do not need to be
on the installation image.
It's also possible to create live images with FAI.
It is a little bit more…
Currently, you need some manual work but that should be also possible
to use FAI for creating live image and then also to provide this
on the FAI.me web service.
If you want to customize much more inside the image, you just say
"Oh, I have some ansible scripts that I want to execute at the very end"
then I say "Ok, this is just a starting point, use the FAI.me service
and if you're happy with the FAI tools, then set up your own FAI server,
create your own configuration space and then you can do all the crazy things."
So, how does FAI.me work internally.
We have a web server where there are some CGI scripts and
this is not the build server, so on the web server, you click "Submit"
"Create my image", all the input is validated so you cannot make nasty things
and then the CGI writes or creates a subdirectory and puts files in it,
a config and a meta file
and writes a status "waiting for processing".
Then, the other server, the build server reads this config and
this is just an NFS mounted directory, and sees
"Oh there's a new job I have to process".
In this processing script we pass for some errors.
What's happening very often that people type in a package that's not available
and this will be detected and then a new version of the web page will pop up and say
"Oh, when creating the package mirror, there was an error
because this package was not known."
Sometimes I have to…
Every night, I create new nfsroots for Buster.
If there are security updates, I have to create new nfsroots
for Stretch and backports.
I have some cleanup, so if a lot of jobs are created,
the images are on the disk after,
normally I say after one day I just remove the images
so you have one day to download the images.
There's 3 different configurations
/etc/fai-stretch, /etc/buster, /etc/fai-stretch-bpo (backports)
We need for the installation image
We need a different nfsroot, but the config space that is shared
about all configurations,
so it doesn't matter if I install
Stretch or Stretch backports or Buster,
I can use the same FAI configuration.
Also, for building the cloud images, I use the same FAI configuration.
A new job is detected, then a copy of the configuration space will be made
and it will be customized a little bit.
So there are a very very few changes,
for example I have to put the SSH key into your customized configuration space
or the list of packages or the user and root password.
Then we have two things, if we want to create the installation image,
I first have to create the partial package mirror and then create the installation image
For the cloud images, we do not need the nfsroot, we just need
the configuration space which is customized a little bit
and then we can just create the disk image
so there's one step less compared to creating the installation ISO.
The status on the web page will be updated, log files written
and if the user said "Please send me an email if my job is ready",
this will also be sent to the user.
The we have the ISO or the disk image and this will be copied back
to the web server where the user can then download it.
And since I have a lot of RAM in this machine,
everything is run in RAM, very very nice.
As I said, we need an nfsroot, a configuration space and FAI classes.
This is a very central component in FAI
and this is just a list of names.
So in HOME_LVM, this is the class name, the FAI class we describe
and I think this is that example:
HOME_LVM describes how to partition the local hard disk.
This is our very flexible tool where we can do LVM, cryptsetups,
software RAIDs and so on.
But for the FAI.me service, I just created 4 different types of partitioning
and this is the HOME_LVM example.
So we have a list of classes and, as I said,
just two commands for the installation image with a list of classes
and for the cloud image, I have to say how big should the disk image be,
the list of classes and what's the target file that should be created.
Let's see if this is ready.
Yes, it's ready.
So…
It's 1.1GB.
Is this really the… oh yeah, raw.
No problem, let's download it, it should be fast.
This is the normal architecture if you use FAI in a client/server set up.
You should just look on the left side where you see
you need the config space, an nfsroot and a mirror
and these parts will put onto the CD.
If you ??? network installation thing,
this is of things get from the server to the client.
For the software installation, we have another subdirectory called package_config
and there you also see several files where the file name is a FAI class.
Since in the FAI.me service every client belongs to the class DEBIAN,
it will install the packages that are listed on the ???
and here we have an other class, NONFREE
These packages are only installed if you also said
"Please install the nonfree packages"
and this is mapped to a FAI class called NONFREE.