Welcome back, the next talk will be Jan Kiszka
on Getting more Debian into our civil infrastructure.
Thank you Michael.
So my name is Jan Kiszka,
you may not know me, I'm not a Debian Developer, not a Debian Maintainer.
I'm just an upstream hacker.
I'm working for Siemens
and part of the Linux team there for now 10 years actually,
more than 10 years.
We are supporting our business units in getting Linux into the products successfully
for that long time, even longer actually.
Today, I'm representing a collaborative project that has some relationship
with Debian, and more soon.
First of all, maybe a surprise to some of you,
our civilization is heavily running on Linux and you may now think about
this kind of devices where some kind of Linux inside,
or you may think of the cloud servers running Linux inside.
But actually, this is about devices closer to us.
In all our infrastructure,
there are control systems, there are management systems included
and many many of them run Linux inside.
Maybe if you are traveling with Deutsche Bahn to this event these days,
there was some Linux system on the train as well,
as they were on the ???, so on the control side.
Energy generation.
Power plants, they are also run with Linux
in very interesting ways, in positive ways
Industry automation, the factories, they have control systems inside
and quite a few are running Linux inside.
And also other systems like health care, diagnostic systems.
These big balls up there, they're magnetic resonance imaging systems,
they're running on Linux for over a decade now.
Building automation, not at home but in the professional building area.
Actually, as I said, the train systems are going to be more on Debian soon.
We have Debian for quite a while in power generation.
"We", in this case, Siemens.
We have the box underneath, on the third row,
the industrial switch there is running Debian.
And the health care device is still on Ubuntu, but soon will be Debian as well.
Just to give some examples.
These are the areas where we, as a group, and we, as Siemens, are active.
But there are some problems with this.
Just take an example from a railway system.
Usually, this kind of devices installation, they have a lifetime
of 25, 30 years.
It used to be quite simple with these old devices,
simple in the sense that it was mechanic, it was pretty robust
I was once told that one of these locking systems,
they were basically left in a box out there for 50 years and no one entered the ???
No one touched the whole thing for 50 years
These times are a little bit over.
Nowadays, we have more electronic systems in these systems
and they contain of course software.
What does it mean?
Just to give you an idea, how this kind of development looks like in this domain.
So ???
development takes quite a long time until the product is ready,
3 to 5 years.
Then, in the railway domain, it's mostly about customizing the systems
for specific installations of the railway systems,
not only in Europe, they are kind of messy regarding the differences.
So you have specific requirements of the customer, the railway operators
to adjust these systems for their needs.
And you see by then,
after 5 years already, a Debian version would be out of maintenance and
if you add an other year, you can start over again.
So, in the development time, you may change still the system
but later on, it's getting hard to change the system ???
because then the interesting parts start in this domain, not only in this domain,
that's safety and security assessment and approval for these systems.
And that also takes time.
For example, in Germany, you go for the Eisenbahn ???
and you ask to get a permission to run that train on the track
and if they say "Mmh, not happy with it", you do it over again
and it takes time
and if you change something in the system, it becomes interesting
because some of these certification aspects become invalid,
you have to redo it.
And then of course, these trains on the installation,
the have a long life as I mentioned before.
So how do you deal with this in an electronic device and
in software-driven devices over this long phase?
That's our challenge
and just one example and there are more in this area.
At the same time, what we see now is these fancy buzzwords
from cloud business entering our conservative, slowly moving domain.
We talk about IoT, industrial IoT, so connected devices.
We talk about edge computing, it means getting the power of the cloud
to the device in the field, closer to where the real things happen.
So, networking becomes a topic.
In the past, you basically built a system, you locked it up physically
you never touched it again, except the customer complains that
there were some bug inside.
These days, the customer asks us to do a frequent update.
And actually the customers ??? ask for this.
So you have to have some security maintenance concept in this
which means regular updates, regular fixes
and that is of course ??? for this kind of doing the way you have
slow running and long running support cycles.
To summarize, there's a very long time we have to maintain our devices in the field
and so far, this was mostly done individually.
So each company, and sometimes quite frequently also inside the company,
each product group, development ??? did it individually.
So everyone was having their own kernel, everyone was having their own base system,
it was easy to build up so it should be easy to maintain.
Of course it's not.
This was one thing, one important thing.
And then, of course, we not always are completely happy
with what the free software gives us.
There are some needs to make things more robust,
to make things more secure, reliable.
So we have to work with these components and improve them, mostly upstream,
and that, of course, is not a challenge we have to address in this area.
And catch up with a trend coming in from the service space on the cloud space.
So with this challenge…
it was the point where we, in this case, a number of big users of
industrial open source systems,
came together and created a new collaborative project.
That's what you do in the open source area.
This project is called Civil Infrastructure Platform.
It's under the umbrella of the Linux Foundation,
there are many projects of the Linux Foundation you may have seen,
but most of them are more in the area of cloud computing
or in the area of media.
Automotive computing, this one is actually even more conservative than the other ones
and it's also comparably small.
Our goal is to build this open source base layer for these application scenarios
based on free software, based on Linux.
We started two years ago.
That's basically our structure, to give you an idea.
Member companies, the 3 on the top are founding platinum companies,
Hitachi, Toshiba and Siemens.
We have Codethink and Plat'Home on board,
we had them on board for the first time as well.
Renesas joined us and just recently also Moxa.
So if you compare this with other collaborative projects,
it's a pretty small one, comparatively small one,
so our budget is also limited.
It's still decent enough, but, well, we are growing.
And based on this budget, we have some developers being paid,
Ben is paid this way, you will see later on why.
And we have people working from the companies in the communities
and we are ramping up on working with communities
to improve the base layers for our needs.
Everything is open source, we have a GitLab repo as well and
you can look up there what's going on there.
So, the main areas of activities where we are working on right now.
4 areas.
Kernel maintenance,
we started with declaring one kernel as the CIP kernel to have
an extended support phase for this kernel of 10 years.
This is what we're aiming for, which is feasible already
for some enterprise distros in a specific area
but here we are talking about an industrial area, an embedded area
so there is some challenge.
I'm saying 10 years, there's sometimes written 15 years,
we will see after 10 years if we follow on to this.
Along with this, of course, comes the need for real time support.
Currently, it's a separated branch, but it's going to be integrated eventually
to have the PREEMPT_RT branch ??? doing this.
As I mentioned before, Ben is currently our 4.4 CIP kernel maintainer.
This is the core, basically where we started activities.
We continued in extending this on test infrastructure,
so we invested a bit in improving on ??? infrastructure,
we are now ramping up an internal ??? just to enable
the kernel testing of course.
And then, that's actually what I'd like to talk about today a bit more,
there's a CIP core.
The kernel alone doesn't make a system, you need a user space,
you need a user land and that's basically where we are now focusing on,
ramping up.
Our activity is to define this CIP core, means a base system,
user space base system which you want to maintain as long as the kernel,
so an other 10 years thing.
Our group had a couple of members which were already familiar with Debian before.
So it was pretty easy for that group to decide on
choosing Debian as the base source for our core, CIP core package.
So, why was Debian chosen?
Well, it has an outstanding maturity and a focus on stability,
so we are pretty much aligned regarding how conservative we see certain things
which is a positive thing for us.
It has very professional security properties but we also rely on heavily.
And also another interesting aspect for us is the license hygiene that you are after
to ensure that there is only free software in these packages
and that is properly documented.
We, when we are using and redistributing software,
in contrast to, for example, the service space
when you don't usually redistribute things,
we are redistributing devices, so we are redistributing software,
we have to take care of the licenses that we are redistributing
and that we are compliant with all these licenses included.
So it's very important for us that this is a consistent picture we get from the package.
Someone looked at this already, we are still looking ourselves on this
but that's a very important thing.
With these characters, we chose Debian as the base system.
So, what does it mean right now?
We are currently in the process to select the core packages from the Debian packages
There is still a little bit of ??? obviously.
So we are already working with Debian on certain long term support aspects
Just to mention 2 activities,
we were sponsoring already the staging repo for security master.
Actually I'm ??? aware of the current state of the project
but we got the feedback that it's apparently a valuable thing for LTS activity
We just joined LTS platinum sponsoring and we are now involved in discussion
for this extended LTS activity,
so anything beyond 5 years
and in the end, that's what we committed to our users.
We want to ensure that for the base system the 10 years is reached.
Of course, ideally, in the community, not only based on our personal activities
but in the end, we have to fill the gap
and that's basically our commitment on this.
Don't take literally what is written here.
This is basically to reflect the package set we are discussing
and there are some 30 to 300 packages on the discussion, so to say right now
We're condensing basically all the input from our users, from our members,
what they are using already
and there is a difference we will later on where this comes from
in the amount of packages, if the way they're using.
So, the kernel currently is not part of the Debian thing we import,
although some of our users would directly use a Debian kernel
but as I said, when there's a need for additional activities and
that's why CIP Core comes in
but then we have a set of base packages
and then of course, we also have to have a certain set of packages that we need to keep
in a usable way to ensure reproducibility of this base set.
Because if we want to fix something after 9 years in the field
on a base system produced in the past,
we have to ensure if we can come up with the same result
plus the delta.
So there are different ways how to build a system
and compared to the classic installation
you may know from a desktop or a server you're not installing,
we are prebuilding images and then deploy these images on the devices
either in the factory or out there in the field.
So the challenge for us is, if we have this package list,
how to get to the device image.
So just to give you a brief idea, of course there is some input
from the CIP kernel in source form
then we are using ??? prebuilt binary packages from Debian
and/or source package, the source feed from Debian,
the upstream source but the Debian patches as input feeds
and that comes bound to a minimum base system to be generated
and we are currently working on this.
There is no defined way of producing this image within CIP at this point,
we are basically following two paths.
One of them is the path which is dominated by the idea
"Ok, we have to ensure we, in this case the ??? environments
have to ensure to reproduce the image ourself, the binaries ourself"
so we take the maintain sources from the Debian community
but we rebuilt and then generate a new binary ??? out of this.
That's one way and that's an activity which you have heard about,
meta-debian project prominently driven by Toshiba,
which uses the ??? way of producing a base system
but out of Debian sources so that you have a maintained source input feed
for this production.
That's one path.
The other path is using predominantly binary packages
and personally and specific also at Siemens we are more following this path here.
So there is for example the ISAR project,
??? is one of their developers here as well
We are working on this path, it means that 95 or 99% of your image consists originally
of binaries, Debian binaries as they are shipped, as they are released
and then there is often the need to modify a little bit,
it might be the kernel, it might be the bootloader,
it might be a special patched package for whatever reason,
hopefully good ones.
You have an infrastructure to assemble the binary images and
to produce the source packages on demand
and install that into an image that you then can flash on the device.
That's the second path we are following, as I said,
that's just to describe the workflows, the technology behind it is
not yet standardized in the CIP.
For us at Siemens, currently, ???
it's also ??? based
yocto-like production,
but based on the Debian binaries producing a ready-to-install device image.
We look at the situation.
So what is Debian providing?
Well, a large set of packages, a nice level of support, 3 + 2 years LTS mostly.
That's already great, I mean there's everything available,
almost everything in the world of Free Software, we can get via Debian.
The build, it supports native build.
That's also an advantage, because finding after 10 years, 15 years with cross build…
There's always a problem with cross building, even a little bit.
So this is a good strategy to go, although you're also working on cross build
that may be interesting for certain scenarios as well for us
and we're all discussing this these days,
reproducible builds is also very important for us
because we also have to prove that the delta is really only on the delta
that has to be changed and not anything else and
we have to rebuild something for whatever reason,
we don't want to produce a completely different image in the end.
So it's a very important topic.
I mentioned already before the license compliance topics.
I'm not really deep expert on all the licensing thing,
except when I have to be because some customer asks us internally
how to be compliant and how to solve certain compliance findings.
A colleague of mine, ??? example who's maintaining the fossology project
is way more in this because we have also our infrastructure
to ensure license compliance and identify packages, ???
and the idea, as far as I heard, is to combine these kinds of activity
so that Debian can also use the information that this kind of scanners produce
like spdx formats and build it into the Debian 5 next generations.
In turn, we can extract this information and ensure that they are still valid
when we take a package.
So there's a lot of activity already in this area
and of course testing, not to mention.
So, what we need to require here, as I said.
One thing is we will need a longer support phase.
The number of packages fortunately is much lower.
As I said, something like 200 at most is what we're currently heading for
for most of our devices.
We have the need to both build natively and cross build predominantly
in the development phase,
but there might also cases where it might be useful for a product image
but predominantly it's for development phase, you want to
??? when you are building on on x64 ARM for example.
The binary source packages should be managed and reproducible.
The license compliance already mentioned.
And the testing activity is also something that we want to improve on further.
So, where we see the collaboration.
Already mentioned long term maintenance for packages,
that's definitely an area where we are reaching out and we are in discussion.
Contributing to Debian cross, there's activities going on this area.
Reproducible builds, we had some discussion, Holger and Chris, these days
where we could possibly support you on this.
It's not our topmost priority at this point but it's obvious that
it will become in the future.
Also, a way possibly interesting for you,
I think there is a good chance that these activities also open up
more adoption in the ??? of Debian.
Because we are also discussing this kind of things with our suppliers,
means the silicon vendors, pushing them to be more upstream
in order to have it easier for us to integrate their work in our systems
and eventually, also enabling them to use the same mechanism that we are using
for building our images to build there our customer SDKs
or however they call them
and that can create a large ecosystem.
We have been discussing already with some of these vendors
and some are actually interested in Debian as well as a default image
to replace those not so successful source build approaches
that are out there in the field eventually with something more easy to use
An other area I really like to see that we have collaboration on
is regarding the license result.
We, at Siemens, currently are running through with this subset package
that fossology run
and I would like to see the result of this run, comparing it to what
Debian is currently reporting in the metadata
if there are any gaps, anything that our experts say
"Ok, you should document it more that way" or "There is something missing"
and of course report these issues upstream
because eventually, I don't want to rescan every single security update package
internally again if you did already.
That should just run through and we should have the trust that
this information is accurate and we can rely on that.
That's the vision behind it.
Test cases would be also an area where we see a chance to contribute something.
Further things we are discussing might be not that interesting for Debian,
but it's interesting in general.
Functional safety activities, you'd be surprised how many people are asking for
functional safe Linux these days,
may it be for automotive, but also for industrial purposes.
Worth mentioning, actually, is the security standard this way.
So even if you're not involved in all this IEC whatever stuff,
it's interesting because this is pushing us, in industry,
to do things like update strategies even more consistently
and ensure that the image that we ship is integer,
so that it's really the original image.
Up to the questions of how to secure the boot
and how to secure the system is running.
That helps us to argue internally and externally for consolidation
and that helps us currently to push a lot of these users
towards Debian solutions.
One of our units did once a survey, recently actually,
about how many Linux systems they have there, and they counted 99 balloons
erm, Linux systems, actually
and of course you can imagine it's pretty hard to maintain 99 variants
in the field out there.
So they are one of the most prominent drivers inside our company
to consolidate the systems and we are currently consolidating over Debian.
Not everything, but most of it.
And then there is this doomsday date as well,
which is an increasing concern because you can imagine that
if you are building a device today, maybe it's out of business in 10 years,
Ok you're lucky, maybe it's still running in 20 years and it's not yet ready for 2038
and then we have a problem.
That's things ??? going on currently already,
so one ??? for example is sponsoring activities in glibc to prove the topic
and as a consortium, the CIP group be also looking into this,
we would not jump in on things but ??? happening
but if there are gaps up there, then we will possibly jump in here as well.
So, to summarize.