FAI.me - A Build Service for Installation and Cloud Images

Talk given by Thomas Lange at Minidebconf Hamburg 18
https://meetings-archive.debian.net/pub/debian-meetings/2018/miniconf-hamburg/2018-05-20/fai.me.webm

Today, I talk about FAI.me, which is a
build server for images.

First, anybody that never heard anything
about FAI?

Ok

I started this project in 1999.

I'm not sure…

No, I'm sure that during those times, the
Debian installer did not have

the preseeding stuff, so we needed
something automatically.

I installed the first cluster with FAI and
I always do talks on FAI or

today in the lightning talks, I talk
a little bit about dracut,

which is used in FAI.

So, what was the motivation.

A neighbour of mine, she came to me with

"My Windows desktop is broken,
can you reinstall it?"

And in the end, I installed her Linux,
and I was shortly thinking about

"Should I use FAI for installing her
desktop with Linux?"

And in the end, I did not use it because
FAI is too complicated,

like the Debian installer, I guess it's
not really that easy for beginners

because there are a lot of questions

but also FAI is not really for beginners.

So this was the motivation about thinking
about FAI.

The target group was always advanced
sysadmins

but I thought maybe it's possible to make
FAI usable also for people

that are not that advanced sysadmins.

The idea is that an installer should cover
most installations.

The Debian installer is really perfect
because I think it covers

all different kinds and strange environments

You can do a lot of things, you can configure
very strange combination of language,

keyboard layout and so on

but I was thinking about an installer
that covers 90 or 95% of the installations

A lot of special cases can be ignored and
since the Debian installer has like

more than 20 questions, I thought it would
be much nicer if there were only

3 to 5 questions and I looked at Linux Mint
and Mageia installers, CentOS installer,

and they all ask much less questions.

In the Debian installer, we sometimes
have also things that are asked

during the installation, so not everything
is asked at the very beginning.

For example, the task selection, where you
select your desktop,

is done after the base installation.

This was also very important, I would like
to have something that

asks everything at the very beginning.

Then, maybe some tool could create
a customized installation image

and this installation image should run
then completely unattended

so you can get yourself a coffee and
when you come back, your machine is ready.

There are 3 things to customize installation
image,

you just put this image, you do not have
to touch anything, and then it's ready.

I thought "Oh yes, this is FAI, maybe
FAI can do this."

As I said, FAI is only, or was until now
only a tool for experienced sysadmins

and you have to adjust several config
files, these are ASCII files

but still you have to touch 5 to 10
config files to make a customization.

So, how can I make FAI usable for
beginners?

That's the beginning of FAI.me.

There's a web page, we'll show it
in more detail later,

where you can just click some things, and
then you get a customized image.

This image can be put onto a CD, DVD or
USB stick, just with dd

and the customization is just by using
the web interface

so there's no need for you to edit
a text file, a config file inside FAI.

I hope I covered most important things
that you want to adjust

or a little bit customize.

You can add additional packages, I think
that's the most important thing

that people say "I want to have the normal
Debian installation

but with some additional packages."

And you can select different
distributions, so it's not only

the installation image for the stable
release, you can create

3 variants of the installation.

This is the web page and thanks to Juri,
he did a great job

during the first and second day, he added
a new feature that we now have

a toggle button.

Is it big enough or should I zoom in?

Ok.

So, we have a toggle button, what you see
now is just the bare minimum of questions

and we can toggle it to more advanced
settings.

You have to select or just leave this as
it is, username,

if you do not enter a password, a password
will be generated and shown to you

and sent by e-mail.

I will now just type in the password.

It's here in clear text, for me that's fine
because

there's also a comment that you should
change the password after the installation

and I do not like to enter passwords twice
so you can see what you typed in

and hopefully do not make any wrong
mistakes.

For example, we could select the Stretch
distribution with backports,

so we will get a 4.15 kernel with Stretch.

There are some buttons we can say we want
to have some Debian developer tools.

This is what I defined in the FAI
configuration, so just a list of packages.

Here, you can enter you own packages.

I will select the desktop.

You can have an installation without any
desktop, so a very small installation.

I will select the XFCE desktop, but all
the other desktops are here.

The language, these are just task packages
that are…

I think Debian has much more task packages,
I just searched which are

the most common languages, and what I do
if I say I want the spanish language,

also the keyboard layout is spanish.

I know there are different combinations
and with local time,

it's getting more difficult.

This installation will install the clock
with UTC, so if you want to set

your time, you have to do this manually.

I want to cover the most common installations.

We select english US, the desktop and,
as an example, the midnight commander

and GIMP.

I can add an email address so if it would
take longer,

for example if this service will have
success and a lot of people are using it,

you may wait for some minutes so your job
will be finished.

So here are the comments, how to reconfigure
the keyboard or the timezone

and then you just click "Create
the installation image".

Now, in the background, there's some job,
a script, looking "Oh, there's a new job"

and there's a summary of the configuration,
of the web configuration.

Down here you see these are the
FAI classes,

I will explain a little bit more about this.

But with this information, FAI configuration
is generated,

that's what normally the experienced
sysadmins have to create

but here you just click on some buttons
and it will be done for you.

In the meantime, we have some more
advanced features

which I will also show you later.

For example, this very simple installation
just creates one partition

but you can also select that you want
to have a separate /home partition

or using lvm just by selecting this
on the web interface.

You can also add your SSH public key
for logging as root without a password

or what's very nice, I found the new
Ubuntu installer does this,

you can give your github account and
then there's a command which

receives the public key from your
github account and puts it

into the root account so you can log in
without password.

I think that's very neat.

And if you have a repository with your own
packages, you could also add this and say

"Please install those packages from
my publicly available repository."

Let's see.

As we see, this job finished in 74 seconds.

Now, this customised installation image
is available for download.

You can also download the log file.

Since this is an installation image,
I first have to create

a partial package mirror.

This is done by the command 'fai-mirror'
and you can also read the log of

this call of the fai-mirror, where a list
of all you packages with all the dependencies

are available.

So you see, these are the list of packages
and later they are downloaded

and in the end, it says it created a mirror
of 1G of packages

and since I have a local mirror,
it's very fast.

This is the one part on the installation
image partial mirror with all the packages

and the other is that the config space
which you can also download.

So this is the config space that was
really created for you

by clicking the web interface.

If you want to do more things with FAI,
you can set up your own FAI server

and use this configuration space.

And, that's also very new, the two commands
that are used for creating this ISO image

are now also listed there.

First, create the partial mirror and then
create the installation image.

Ok, copy link location…

Let's see how good the network is here.

[Q] It's a rather large image.

Yeah, because it includes all the packages
and with Xfce, LibreOffice and so on

and the installation environment is maybe
about 200MB.

That's not much bigger than the Debian
installer that you need to download.

So, 2, 1, done.

I have a little wrapper which calls
a fresh kvm machine

with an empty disk and boots this ISO image

and then we will see how this installation
runs.

So this is dracut booting the image

and now you see there are already
some parted commands executed

and now the packages are installed
and everything runs on

and in the end some customization
script.

We use only shell scripts for doing
some customizations

and you see the files are downloaded
from /media/mirror

so this is local on the ISO image.

It would also be possible to create
an image without the packages

and then give another sources.list file
so the packages would be downloaded

from the internet but this default
in the FAI service,

we put everything onto the ISO image.

I guess it will run for 4 minutes.

[Q] ???

Yeah.

What I will show you now is…

So, this was the simple one, now I toggle
this web page and

you will see that there are some more
questions you may answer,

for example you can give a root password.

If you leave this empty, sudo will be
configured.

Here you can upload the SSH key or give
your Github account,

that would be Mrfai for me.

With the partitioning schemes, we have
one partition

or one partition and /home separated
or these two versions with LVM.

FAI itself can do much more, we could do
soft raid set ups, cryptsetup

but here I want to cover the most common
installation, so very simple

we have only 4 things that you can choose.

[Q] For encryption?

[A] Yes.

So, this was the partitioning things.

This is the new feature where you can add
an URL for your local package repository

and the rest is the same, you can add
packages you like, your email address

and then also create an installation image.

I normally set… By default, I include
the nonfree linux firmware.

This is because my target audience is
an end user and I want to make it

very comfortable for them, so yeah,
they can just install it

and do not have these problems.

And since this is not on an official
debian.org web site,

I can do this with this default.

Let's see, the installation is still running.

So, advanced features.

The next thing after this installation,

I will show you how to create cloud
images.

Currently, we create an installation image,

when you boot it, the installation is run
fully automatically.

The other type of service FAI.me gives
to you is that it creates a raw image

or some other formats as you see here,
qcow2 and whatever

which you can just boot and
the installation is already done.

But first, see if the installation finish.

Ah! Ok.

These are now the shell scripts that are
executed for the customization

of like /etc/messages of today,
/etc/network/interfaces is written

and so on.

You see the installation took 236s, it says
there are some errors

but that's not really true.

And it stops here, but we can also
disable this, this is only for showing

everything went well

and now we just reboot the machine.

You see the grub.

Ok, Xfce desktop.

debian was the user with password FAI.

We have "uname -a", this is 4.16,
the backports kernel was installed there.

We have only one partition, no LVM

and I told it to install gimp which is
not installed by default.

Gimp is there, so this is nice.

And the midnight commander is also there.

And now we just throw this machine.
Gone.

What's very nice with this wrapper script,
it creates the local disk

of the virtual machine in /tmp which is
a RAM disk and I love RAM,

it's so nice and fast.

So, this was installation image and now we
look at the cloud image.

First, you can say how big should your
disk image be.

Here, I say 8GB, you will see it's not
an 8GB image

that you have to download later.

By default, I use zstd compression.

Anyone who does not know
this compression?

This is very fast, very new, created
by Facebook if I'm correct.

It's for very big files and what you should
never use is gzip with sparse images.

The disk image is sparse and gzip
cannot handle this

so if you compress it and uncompress it
it will be very large

and all the other, xz, zstd, can handle
sparse files very nicely.

So, the hostname is set, the root password,
username with a password.

Now we want to install Buster.

Maybe with no…

Oh, we also do the Xfce desktop.

Any packages you'd like to have in this
cloud image.

"desktop" and "cloud" image does not
make that much sense, maybe.

Emacs25, ok.

And now "Create disk image".

This will take a little bit longer because
we are doing the installation

inside a file image.

But no problem, I can tell you what
other ideas I have.

So, currently we have the installation and
the cloud or virtual machine images for amd64

FAI itself can also do cross-architecture
images so it would be some work

to extend the web page to say
"Please create an arm64 image"

It would be very nice to have predefined
configs for raspberry pi or

all the very different boards

but that would also be possible.

I guess the next thing I will implement
is other distributions

because I know people are always asking it.

Not you but the Ubuntu guys.

Yesterday I did the first test with Ubuntu
bionic, the LTS release

and FAI just works out of the box with it.

So what I have to do is to integrate it
in these FAI.me processing scripts.

Ready-to-go cloud images for the big
cloud providers.

That's only a different FAI config space
that I have to use.

Currently, for example, in what I call
cloud images, I do not install

the package cloud image.

That's needed for all the ones.

I'm also working in the Debian cloud team
and this team decided 2 years ago

that the tool chain in the future for
the official Debian package will be FAI.

Amazon is already using it, so if you
boot or if you use a Debian cloud image

in Amazon, Noah Meyerhans did this and
he's using the FAI tool chain for it.

Google is not yet using it because there was
a very small problem

in a config file we had one space too much

which caused grub to hang forever

and that was the reason why they decided
for Stretch to use their own tool chain.

But the things are working so we have
the config space also for Google.

And also for Azure, some people from
Credativ did this.

The Debian cloud team already has
the FAI configuration for

the big tool providers,
cloud providers.

We could also think on a more generic
FAI installation image.

It's an image that you would boot up
and then enter your job id of the web page

and then the configuration would be
downloaded

and the packages would be just
received from the internet.

That was one…

So, the image would be much smaller
because the packages do not need to be

on the installation image.

It's also possible to create live images
with FAI.

It is a little bit more…

Currently, you need some manual work
but that should be also possible

to use FAI for creating live image
and then also to provide this

on the FAI.me web service.

If you want to customize much more
inside the image, you just say

"Oh, I have some Ansible scripts that
I want to execute at the very end"

then I say "Ok, this is just a starting
point, use the FAI.me service

and if you're happy with the FAI tools,
then set up your own FAI server,

create your own configuration space
and then you can do all the crazy things."

So, how does FAI.me work internally.

We have a web server where there are
some CGI scripts and

this is not the build server, so on the
web server, you click "Submit"

"Create my image", all the input
is validated so you cannot make nasty things

and then the CGI writes or creates
a subdirectory and puts 2 files in it,

a config and a meta file

and writes a status
"waiting for processing".

Then, the other server, the build server
reads this config and

this is just an NFS mounted directory,
and sees

"Oh there's a new job I have to process".

In this processing script we pass for
some errors.

What's happening very often that people
type in a package that's not available

and this will be detected and then a new
version of the web page will pop up and say

"Oh, when creating the package mirror,
there was an error

because this package was not known."

Sometimes I have to…

Every night, I create new nfsroots
for Buster.

If there are security updates, I have
to create new nfsroots

for Stretch and backports.

I have some cleanup, so if a lot of jobs
are created,

the images are on the disk after,

normally I say after one day I just
remove the images

so you have one day to download
the images.

There's 3 different configurations

/etc/fai-stretch, /etc/buster,
/etc/fai-stretch-bpo (backports)

We need for the installation image

We need a different nfsroot, but
the config space that is shared

about all configurations,

so it doesn't matter if I install

Stretch or Stretch backports or
Buster,

I can use the same FAI configuration.

Also, for building the cloud images,
I use the same FAI configuration.

A new job is detected, then a copy of
the configuration space will be made

and it will be customized a little bit.

So there are a very very few changes,

for example I have to put the SSH key
into your customized configuration space

or the list of packages or the user and
root password.

Then we have two things, if we want to
create the installation image,

I first have to create the partial package
mirror and then create the installation image

For the cloud images, we do not need
the nfsroot, we just need

the configuration space which is
customized a little bit

and then we can just create the disk image

so there's one step less compared to
creating the installation ISO.

The status on the web page will be
updated, log files written

and if the user said "Please send me
an email if my job is ready",

this will also be sent to the user.

The we have the ISO or the disk image
and this will be copied back

to the web server where the user can then
download it.

And since I have a lot of RAM
in this machine,

everything is run in RAM, very very nice.

As I said, we need an nfsroot,
a configuration space and FAI classes.

This is a very central component in FAI

and this is just a list of names.

So in HOME_LVM, this is the class name,
the FAI class we describe

and I think this is that example:

HOME_LVM describes how to partition
the local hard disk.

This is our very flexible tool where
we can do LVM, cryptsetups,

software RAIDs and so on.

But for the FAI.me service, I just created
4 different types of partitioning

and this is the HOME_LVM example.

So we have a list of classes and,
as I said,

just two commands for the installation
image with a list of classes

and for the cloud image, I have to say
how big should the disk image be,

the list of classes and what's the target
file that should be created.

Let's see if this is ready.

Yes, it's ready.

So…

It's 1.1GB.

Is this really the… oh yeah, raw.

No problem, let's download it, it should
be fast.

This is the normal architecture if you use
FAI in a client/server set up.

You should just look on the left side
where you see

you need the config space, an nfsroot
and a mirror

and these parts will put onto the CD.

If you ??? network installation thing,

this is of things get from the server
to the client.

For the software installation, we have
another subdirectory called package_config

and there you also see several files
where the file name is a FAI class.

Since in the FAI.me service every client
belongs to the class DEBIAN,

it will install the packages that are listed
on the ???

and here we have an other class, NONFREE

These packages are only installed if
you also said

"Please install the nonfree packages"

and this is mapped to a FAI class
called NONFREE.

And there's an other class for AMD64
and so on.

Some references.

In the past, it looked more like this
when I said

"Oh, who's using FAI?" and during
the last month I collected some logos

just because it's much nicer.

Let's see if the download was ready.

We unzstd the FAI.me image,
faime-013Z image

On the web site, I said I want to have
a 8GB partition,

so now let's see how big it is.

The file is 8, but since it's a sparse file
it's only 3.5GB

and the compressed was 1.1GB.

Now I use my wrapper

and I say "Boot from disk" and this is
the FAI.me raw image, disk image

that should be booted up.

That's it.

debian/fai

Let's see if emacs is installed, yes.

Gimp is already there, hopefully, and
the blue midnight commander.

Let's see.

Questions.

[Q] I'm using the preseed file for
the debian-installer,

do you have a conversion between
your syntax and your configuration files

and the preseed file or maybe can you add
a download button for the preseed file

to your web site because I think it's
rather nice to have it displayed

in web site first.

[A] I'm not using the debian-installer.

I use preseeding, yes, the debconf
preseeding for the normal packages

you can do this also in FAI and it's
the same format

you get with debconf-get-selections.

And what you get is you can download
your own FAI config space

and this includes all information you need
to set up,

to do this mirror FAI CD or the FAI disk
image command.

But you cannot convert this config into
a d-i preseeding or vice versa,

that's not possible.

Because for example, for the partitioning
part I do not like to create

from my disk config partman preseeding
file.

You can pay me a lot of money, I will
never do this.

You know that the partman preseeding
is very ugly and very heavy.

For other things, yes, selection of,
for example, the selection of the language

These are the normal preseeding we use.

And the list of packages, task selec…

I think it's much easier to do this in
the FAI configuration than to create

a debian-installer preseeding.

And why use d-i if this works for you?

[Q] d-i works as well for me.

[A] Yes, then fine, use it.

[Q] Hi Thomas. Thank you very much for
this new feature in the FAI project,

it's very nice and I found very great that
you have the output of the commands

that you used to create the ISO image or
the cloud file.

A question that I have is, in which servers
are located the files that we create,

the ISO or the cloud.

Is it a server that you own host or…

[A] Trust me.

Currently,

both the web server and the FAI.me
processing build server are run

on two machines at the university where
I work as a system administrator

so that's also where we have a very fast
connection.

The CGI script and shell script that is
processing these jobs is currently

not open source.

There are plans to do this, I'm not sure
when.

If you want to reproduce the things, you
have the config file and you can download

the FAI software and use these one or two
commands to reproduce it.

Some people said "Oh, very nice service,
I would like to set up in my company".

Then please yes, contact me and…

Currently there are no concrete plans
to make these background scripts open source

but it will be in some future.

But currently, you have to trust me as
you also have to trust the package maintainers

that will be installed there.

But you can verify it or say "I do not
trust Thomas but I will just grab

the FAI config space and this on my own".

[Q] Thank you.

[Q] There's a question from the internet.

Why not use a proper job queuing system
like grid engine or similar?

[A] I'm using grid engine at work for
different things.

It started as a very simple project, so
in the end it's just a loop which

checks if there's new jobs on that.

Currently, I do not process jobs
in parallel, currently there's no need for it

If this project will be very successful,
yeah, I have to use a queuing system.

It's, yeah, a very simple script.

But it would be also possible with
a proper queuing system.

More questions?

[Q] I have a bunch of questions.

First, what is it that you use for
partitioning?

[A] I'm using a poll script that we wrote
several years ago in FAI

and we defined this config file,
this package config

and the Perl script parses this script
and then executes the parted and mkfs command

which you can see in the log files, so if
you want to see what does FAI do

after parsing this, which commands are
executed, you see everything

on the log files.

[Q] Right, but so you turn this text
into partitioning…

[A] commands, yeah.

[Q] But the text looks like this,
like with the spaces and everything.

[A] You can use more or less spaces
or do you like,

should I convert it to XML?

[Q] Ok, then my next question is

what are you using the nfsroot for when
you're generating the images?

[A] The nfsroot is used only for the
installation image.

When I do the installation, I need to boot
the machine as a diskless client,

so it's just what the debian-installer
into RAM,

you need a running Linux system.

This is our nfsroot, on the installation
image.

When you boot installation image, this
nfsroot with all the commands we need

are started without using the local disk
and then we can do

everything on the disk, /root and /target
and so on.

The nfsroot is the system that is running
during the installation.

[Q] Ok, but there's no need for this to be
nfs, it could be a

[A] It's called nfsroot.

This is very common that people call it
nfsroot

and if you have this network installation
thing, it's really an nfsroot.

But you're right.

On the installation ISO, it's not nfs,
it's just a local file system, yes.

[Q] Alright. So, I think it's my last
comment.

You have the ISO from which you install
and when you install from the ISO

you're installing then packages on the
machine, and then you have the image

which is like a disk image that has
the packages already installed,

so you skip the installing step.

Have you thought about having
an intermediate thing

where you download an image that
already has the packages installed?

[A] That's also possible.

When you do an installation, before you
can change root in the new system

for adding packages, you have to call
debootstrap.

What we do, we call debootstrap once and
create a tar file out of it.

This is our minimal… in the former days
it was the floppy disk,

our base tar.gz file,

so you could exchange the minimal tar file
with whatever tar file you have.

That's for example what we do if we install
Ubuntu.

We boot the installation system which is
a Debian system

and then create the local filesystem and
extract an Ubuntu base image

and then we can change root into
the Ubunto or the same for CentOS and so on

then we can change root into the other
Linux system and add packages there.

If you have already a bigger image with
some more packages added there,

it's very easy to say "Do not extract the
Debian Stretch image

but use my image which also
includes other tools."

And if you are fine with that, you can
just extract the tar file.

[Q] Ok. Any more questions?

[Q] The heading is in german.

[A] What?

[Q] The heading is in german.

[A] Oh, because it's a copy of my german
slides.

Thank you for this.

And, what's also missing.

The web page, where you can select german
or other languages,

it would be nice if people are interested
to help translate them

so that it's more easy for people that
do not speak english

to use the website and create their own
installation image with their language.

[Q] Someone on the stream said that
the fai.me web site is not yours

and it's a hack thing, it's a scam, you go
there and get hacked.

Do you have any plans to try to buy
the domain because it's pretty confusing.

The first thing I would have done
by seeing that talk would have been

to go to fai.me.

[A] Yeah, I was thinking about which name
I should choose.

I didn't check which web domains are free
and in then end I thought

"Do I like to have a different domain name
for the service?"

But since it's only a part of the FAI
project, I thought,

and technically it was also easier just
to host it under a subdirectory

and yeah, if people now grab the fai.me
domain and do other things with it, yeah.

I think one question would be to use
a debian.net or maybe debian.org domain

because people trust much more.

I get a lot of comments "Oh this would
be very nice if this would be hosted

on a Debian machine"

but this would be much more complicated
because the DSA team

has much more restriction what to execute
on their machines.

currently, we need root access because
we mount some things

and DSA would not give root access
on any Debian machine.

That's the same problem we have
in the Debian cloud team

where we want to create the official
images for the cloud providers

where the Debian cloud team will also
not have root access

and so there's much more work to get
empty virtual machines

starting up, putting data into it,
creating the images,

receiving them from inside the image.

And since on those machines I have
root access, that's much easier for me.

We are out of time, so thank you Thomas.

[Applause]