Welcome back, the next talk will be
Jan Kiszka
on Getting more Debian into our
civil infrastructure.
Thank you Michael.
So my name is Jan Kiszka,
you may not know me, I'm not a Debian
Developer, not a Debian Maintainer.
I'm just an upstream hacker.
I'm working for Siemens
and part of the Linux team there
for now 10 years actually,
more than 10 years.
We are supporting our business units
in getting Linux into the products successfully
for that long time, even longer actually.
Today, I'm representing a collaborative
project that has some relationship
with Debian, and more soon.
First of all, maybe a surprise to some
of you,
our civilization is heavily running on Linux
and you may now think about
this kind of devices where some kind of
Linux inside,
or you may think of the cloud servers
running Linux inside.
But actually, this is about devices closer
to us.
In all our infrastructure,
there are control systems, there are
management systems included
and many many of them run Linux inside.
Maybe if you are traveling with Deutsche
Bahn to this event these days,
there was some Linux system on the train
as well,
as they were on the ???,
so on the control side.
Energy generation.
Power plants, they are also run with Linux
in very interesting ways, in positive ways
Industry automation, the factories, they
have control systems inside
and quite a few are running Linux inside.
And also other systems like health care,
diagnostic systems.
These big balls up there, they're magnetic
resonance imaging systems,
they're running on Linux for over
a decade now.
Building automation, not at home but in
the professional building area.
Actually, as I said, the train systems are
going to be more on Debian soon.
We have Debian for quite a while in
power generation.
"We", in this case, Siemens.
We have the box underneath,
on the third row,
the industrial switch there is running
Debian.
And the health care device is still
on Ubuntu, but soon will be Debian as well.
Just to give some examples.
These are the areas where we, as a group,
and we, as Siemens, are active.
But there are some problems with this.
Just take an example from a railway
system.
Usually, this kind of devices installation,
they have a lifetime
of 25, 30 years.
It used to be quite simple with these
old devices,
simple in the sense that it was mechanic,
it was pretty robust
I was once told that one of these locking
systems,
they were basically left in a box out there
for 50 years and no one entered the ???
No one touched the whole thing for 50 years
These times are a little bit over.
Nowadays, we have more electronic systems
in these systems
and they contain of course software.
What does it mean?
Just to give you an idea, how this kind
of development looks like in this domain.
So ???
development takes quite a long time
until the product is ready,
3 to 5 years.
Then, in the railway domain, it's mostly
about customizing the systems
for specific installations of the railway
systems,
not only in Europe, they are kind of messy
regarding the differences.
So you have specific requirements of the
customer, the railway operators
to adjust these systems for their needs.
And you see by then,
after 5 years already, a Debian version
would be out of maintenance and
if you add an other year, you can start
over again.
So, in the development time, you may
change still the system
but later on, it's getting hard to change
the system ???
because then the interesting parts start
in this domain, not only in this domain,
that's safety and security assessment and
approval for these systems.
And that also takes time.
For example, in Germany, you go for the
Eisenbahn ???
and you ask to get a permission to run
that train on the track
and if they say "Mmh, not happy with it",
you do it over again
and it takes time
and if you change something in the
system, it becomes interesting
because some of these certification
aspects become invalid,
you have to redo it.
And then of course, these trains on
the installation,
the have a long life as I mentioned
before.
So how do you deal with this in
an electronic device and
in software-driven devices over
this long phase?
That's our challenge
and just one example and there are
more in this area.
At the same time, what we see now is
these fancy buzzwords
from cloud business entering
our conservative, slowly moving domain.
We talk about IoT, industrial IoT, so
connected devices.
We talk about edge computing, it means
getting the power of the cloud
to the device in the field, closer to
where the real things happen.
So, networking becomes a topic.
In the past, you basically built a system,
you locked it up physically
you never touched it again, except
the customer complains that
there were some bug inside.
These days, the customer asks us to
do a frequent update.
And actually the customers ???
ask for this.
So you have to have some security
maintenance concept in this
which means regular updates, regular fixes
and that is of course ???
for this kind of doing the way you have
slow running and long running
support cycles.
To summarize, there's a very long time
we have to maintain our devices in the field
and so far, this was mostly done
individually.
So each company, and sometimes quite
frequently also inside the company,
each product group, development ???
did it individually.
So everyone was having their own kernel,
everyone was having their own base system,
it was easy to build up so it should be
easy to maintain.
Of course it's not.
This was one thing, one important thing.
And then, of course, we not always are
completely happy
with what the free software gives us.
There are some needs to make things
more robust,
to make things more secure, reliable.
So we have to work with these components
and improve them, mostly upstream,
and that, of course, is not a challenge
we have to address in this area.
And catch up with a trend coming in from
the service space on the cloud space.
So with this challenge…
it was the point where we, in this case,
a number of big users of
industrial open source systems,
came together and created a new
collaborative project.
That's what you do in the open source
area.
This project is called Civil Infrastructure
Platform.
It's under the umbrella of the Linux
Foundation,
there are many projects of the Linux
Foundation you may have seen,
but most of them are more in the area
of cloud computing
or in the area of media.
Automotive computing, this one is actually
even more conservative than the other ones
and it's also comparably small.
Our goal is to build this open source
base layer for these application scenarios
based on free software, based on Linux.
We started two years ago.
That's basically our structure, to give
you an idea.
Member companies, the 3 on the top are
founding platinum companies,
Hitachi, Toshiba and Siemens.
We have Codethink and Plat'Home
on board,
we had them on board for the first time
as well.
Renesas joined us and just recently also
Moxa.
So if you compare this with other
collaborative projects,
it's a pretty small one, comparatively
small one,
so our budget is also limited.
It's still decent enough, but, well,
we are growing.
And based on this budget, we have
some developers being paid,
Ben is paid this way, you will see
later on why.
And we have people working from
the companies in the communities
and we are ramping up on working with
communities
to improve the base layers for our needs.
Everything is open source, we have
a GitLab repo as well and
you can look up there what's going on there.
So, the main areas of activities where
we are working on right now.
4 areas.
Kernel maintenance,
we started with declaring one kernel as
the CIP kernel to have
an extended support phase for this kernel
of 10 years.
This is what we're aiming for, which is
feasible already
for some enterprise distros
in a specific area
but here we are talking about an industrial
area, an embedded area
so there is some challenge.
I'm saying 10 years, there's sometimes
written 15 years,
we will see after 10 years if we follow
on to this.
Along with this, of course, comes the need
for real time support.
Currently, it's a separated branch, but
it's going to be integrated eventually
to have the PREEMPT_RT branch
??? doing this.
As I mentioned before, Ben is currently
our 4.4 CIP kernel maintainer.
This is the core, basically where we
started activities.
We continued in extending this on
test infrastructure,
so we invested a bit in improving on
??? infrastructure,
we are now ramping up an internal
??? just to enable
the kernel testing of course.
And then, that's actually what I'd like
to talk about today a bit more,
there's a CIP core.
The kernel alone doesn't make a system,
you need a user space,
you need a user land and that's basically
where we are now focusing on,
ramping up.
Our activity is to define this CIP core,
means a base system,
user space base system which you want
to maintain as long as the kernel,
so an other 10 years thing.
Our group had a couple of members which
were already familiar with Debian before.
So it was pretty easy for that group
to decide on
choosing Debian as the base source
for our core, CIP core package.
So, why was Debian chosen?
Well, it has an outstanding maturity and
a focus on stability,
so we are pretty much aligned regarding
how conservative we see certain things
which is a positive thing for us.
It has very professional security properties
but we also rely on heavily.
And also another interesting aspect for us
is the license hygiene that you are after
to ensure that there is only free software
in these packages
and that is properly documented.
We, when we are using and redistributing
software,
in contrast to, for example, the service space
when you don't usually redistribute things,
we are redistributing devices, so we are
redistributing software,
we have to take care of the licenses
that we are redistributing
and that we are compliant with all these
licenses included.
So it's very important for us that this is
a consistent picture we get from the package.
Someone looked at this already, we are still
looking ourselves on this
but that's a very important thing.
With these characters, we chose Debian
as the base system.
So, what does it mean right now?
We are currently in the process to select
the core packages from the Debian packages
There is still a little bit of ???
obviously.
So we are already working with Debian on
certain long term support aspects
Just to mention 2 activities,
we were sponsoring already the staging
repo for security master.
Actually I'm ??? aware of the current
state of the project
but we got the feedback that it's
apparently a valuable thing for LTS activity
We just joined LTS platinum sponsoring
and we are now involved in discussion
for this extended LTS activity,
so anything beyond 5 years
and in the end, that's what we committed
to our users.
We want to ensure that for the base system
the 10 years is reached.
Of course, ideally, in the community,
not only based on our personal activities
but in the end, we have to fill the gap
and that's basically our commitment
on this.
Don't take literally what is written here.
This is basically to reflect the package set
we are discussing
and there are some 30 to 300 packages
on the discussion, so to say right now
We're condensing basically all the input
from our users, from our members,
what they are using already
and there is a difference we will later
on where this comes from
in the amount of packages, if the way
they're using.
So, the kernel currently is not part of
the Debian thing we import,
although some of our users would directly
use a Debian kernel
but as I said, when there's a need for
additional activities and
that's why CIP Core comes in
but then we have a set of base packages
and then of course, we also have to have
a certain set of packages that we need to keep
in a usable way to ensure reproducibility
of this base set.
Because if we want to fix something
after 9 years in the field
on a base system produced in the past,
we have to ensure if we can come up
with the same result
plus the delta.
So there are different ways how to build
a system
and compared to the classic installation
you may know from a desktop or a server
you're not installing,
we are prebuilding images and then deploy
these images on the devices
either in the factory or out there
in the field.
So the challenge for us is, if we have
this package list,
how to get to the device image.
So just to give you a brief idea, of course
there is some input
from the CIP kernel in source form
then we are using ???
prebuilt binary packages from Debian
and/or source package, the source feed
from Debian,
the upstream source but the Debian patches
as input feeds
and that comes bound to a minimum
base system to be generated
and we are currently working on this.
There is no defined way of producing
this image within CIP at this point,
we are basically following two paths.
One of them is the path which is dominated
by the idea
"Ok, we have to ensure we, in this case
the ??? environments
have to ensure to reproduce the image
ourself, the binaries ourself"
so we take the maintain sources from
the Debian community
but we rebuilt and then generate a new
binary ??? out of this.
That's one way and that's an activity
which you have heard about,
meta-debian project prominently driven
by Toshiba,
which uses the ???
way of producing a base system
but out of Debian sources so that you have
a maintained source input feed
for this production.
That's one path.
The other path is using predominantly
binary packages
and personally and specific also at Siemens
we are more following this path here.
So there is for example the ISAR project,
??? is one of their developers here
as well
We are working on this path, it means that
95 or 99% of your image consists originally
of binaries, Debian binaries as they are
shipped, as they are released
and then there is often the need to modify
a little bit,
it might be the kernel, it might be
the bootloader,
it might be a special patched package
for whatever reason,
hopefully good ones.
You have an infrastructure to assemble
the binary images and
to produce the source packages
on demand
and install that into an image that you
then can flash on the device.
That's the second path we are following,
as I said,
that's just to describe the workflows,
the technology behind it is
not yet standardized in the CIP.
For us at Siemens, currently,
???
it's also ??? based
yocto-like production,
but based on the Debian binaries
producing a ready-to-install device image.
We look at the situation.
So what is Debian providing?
Well, a large set of packages, a nice
level of support, 3 + 2 years LTS mostly.
That's already great, I mean there's
everything available,
almost everything in the world of
Free Software, we can get via Debian.
The build, it supports native build.
That's also an advantage, because finding
after 10 years, 15 years with cross build…
There's always a problem with
cross building, even a little bit.
So this is a good strategy to go, although
you're also working on cross build
that may be interesting for certain
scenarios as well for us
and we're all discussing this these days,
reproducible builds is also very important
for us
because we also have to prove that
the delta is really only on the delta
that has to be changed and not anything
else and
we have to rebuild something for
whatever reason,
we don't want to produce a completely
different image in the end.
So it's a very important topic.
I mentioned already before the license
compliance topics.
I'm not really deep expert on all the
licensing thing,
except when I have to be because some
customer asks us internally
how to be compliant and how to solve
certain compliance findings.
A colleague of mine, ??? example
who's maintaining the fossology project
is way more in this because we have also
our infrastructure
to ensure license compliance and identify
packages, ???
and the idea, as far as I heard, is to
combine these kinds of activity
so that Debian can also use the information
that this kind of scanners produce
like spdx formats and build it into
the Debian 5 next generations.
In turn, we can extract this information
and ensure that they are still valid
when we take a package.
So there's a lot of activity already
in this area
and of course testing, not to mention.
So, what we need to require here,
as I said.
One thing is we will need a longer support
phase.
The number of packages fortunately is
much lower.
As I said, something like 200 at most is
what we're currently heading for
for most of our devices.
We have the need to both build natively and
cross build predominantly
in the development phase,
but there might also cases where it might
be useful for a product image
but predominantly it's for development
phase, you want to
??? when you are building on on x64 ARM
for example.
The binary source packages should be
managed and reproducible.
The license compliance already
mentioned.
And the testing activity is also something
that we want to improve on further.
So, where we see the collaboration.
Already mentioned long term maintenance
for packages,
that's definitely an area where we are
reaching out and we are in discussion.
Contributing to Debian cross, there's
activities going on this area.
Reproducible builds, we had some
discussion, Holger and Chris, these days
where we could possibly support you
on this.
It's not our topmost priority at
this point but it's obvious that
it will become in the future.
Also, a way possibly interesting for you,
I think there is a good chance that
these activities also open up
more adoption in the ???
of Debian.
Because we are also discussing this kind
of things with our suppliers,
means the silicon vendors, pushing them
to be more upstream
in order to have it easier for us to
integrate their work in our systems
and eventually, also enabling them to
use the same mechanism that we are using
for building our images to build there
our customer SDKs
or however they call them
and that can create a large ecosystem.
We have been discussing already with
some of these vendors
and some are actually interested
in Debian as well as a default image
to replace those not so successful
source build approaches
that are out there in the field
eventually with something more easy to use
An other area I really like to see that
we have collaboration on
is regarding the license result.
We, at Siemens, currently are running
through with this subset package
that fossology run
and I would like to see the result of
this run, comparing it to what
Debian is currently reporting in the
metadata
if there are any gaps, anything that our
experts say
"Ok, you should document it more that way"
or "There is something missing"
and of course report these issues upstream
because eventually, I don't want to rescan
every single security update package
internally again if you did already.
That should just run through and
we should have the trust that
this information is accurate and we can
rely on that.
That's the vision behind it.
Test cases would be also an area where
we see a chance to contribute something.
Further things we are discussing might be
not that interesting for Debian,
but it's interesting in general.
Functional safety activities, you'd be
surprised how many people are asking for
functional safe Linux these days,
may it be for automotive, but also for
industrial purposes.
Worth mentioning, actually, is the
security standard this way.
So even if you're not involved in all
this IEC whatever stuff,
it's interesting because this is pushing
us, in industry,
to do things like update strategies
even more consistently
and ensure that the image that we ship
is integer,
so that it's really the original image.
Up to the questions of how to secure
the boot
and how to secure the system is running.
That helps us to argue internally and
externally for consolidation
and that helps us currently to push
a lot of these users
towards Debian solutions.
One of our units did once a survey,
recently actually,
about how many Linux systems they have
there, and they counted 99 balloons
erm, Linux systems, actually
and of course you can imagine
it's pretty hard to maintain 99 variants
in the field out there.
So they are one of the most prominent
drivers inside our company
to consolidate the systems and we are
currently consolidating over Debian.
Not everything, but most of it.
And then there is this doomsday date
as well,
which is an increasing concern because
you can imagine that
if you are building a device today, maybe
it's out of business in 10 years,
Ok you're lucky, maybe it's still running
in 20 years and it's not yet ready for 2038
and then we have a problem.
That's things ??? going on currently
already,
so one ??? for example is sponsoring
activities in glibc to prove the topic
and as a consortium, the CIP group be also
looking into this,
we would not jump in on things but
??? happening
but if there are gaps up there, then we will
possibly jump in here as well.
So, to summarize.