Debian_on_ARM_devices_2.webm

So, I think we are ready to start

So, next talk is from Martin, about Debian on ARM devices.

So, it's Debian on ARM devices, it's not about the Debian ARM port itself, or any of the various ARM ports

I would say it's sort of a mix of different topics

it's basically ---

I mean, basically I often get
it's basically ---

I mean, basically I often get

the question from users

"Well, Debian works on this particular ARM device,

I have a device which is like really really similar,

so surely Debian should work on it, right?"

And so far, the answer unfortunately was usually "no"

and - and it's really frustrating for users

because they don't see why.

You know, the device is very much the same, so why doesn't it just work.

And so, basically what I'm trying to do is

to show like a little bit behind the scenes

of how Debian on ARM works

And there has been a whole lot of progres

So I'm going to show how did it work,

what are the improvements, and

and how is it going to work in the future

And it's all more from a develeoper perspective

so, it's like, how do we support it in the Debian Installer

its' like a mix of stuff, and I should say,

I'm really looking at it from somebody who basically

adds support for devices in the installer

but I'm not like a kernel guy or anything, so

there are a lot of people in this room who know a bit more about the technical details

but I'm not going to really drill down really deep down anyway

But if I'm incorrect, I'm sure that

there are plenty of people that will correct me.

So, where do you find ARM? So it's pretty much everywhere these days, I mean...

Pretty much every phone has an ARM chip in it

All those "Internet of Things" gadgets are ARM-based

You have NAS devices, and that's really the area I used to focus on

those small boxes where you basically add a hard drive

and most people just use it for storage, but it's actually a

full PC where you can install Debian on it

so I always found that to be very interesting

and nowadays you have a lot of development boards

And I'm not sure whether "development boards" is the right word

Because when I hear "development boards" I think of it as really expensive,

Like 5K for boards or something

But nowadays you have those, maybe, a better board would be like bareboards

Like "Raspberry Pi", like basically it's just the board

it does not come with a case, you can buy those as well

and it's like 30 dollars or something, you can get

And that gets pretty much where most of the

excitement is going for Debian users at this moment

And that can cause a lot of frustration

And everybody says that ARM is going to be big on servers

and I think that's something we ar going to see

But right now, again, it's

a little bit frustrating

How does it work as a Deban Installer

There are basically two

ways of installing on those ARM Pi devices

So the, like, historic, that we normally

would be to install on, like, ascreen

can be a serial console,

and then you just download via the network

and then you just download via the network

from a CD image or wherever,

But the other method, which we actually use for those

net devices was a network console

where you basically ssh into the installer

and you perform the installation via ssh

and that was really the only way, because those mass devices

don't have any I/O, so I

even in there, you can

it's just a serial console, most of them

That's something we didn't want to require from the user

And the other thing that's coming through, I think that it's basically cool that was announce today,

screen support, so basically

so you can have multiple sessions with in vi,

which can be useful if you have an open shell to debug

And so, how -- And so, this is really looking back at

like running Debian on NAS devices

which is something that used to be really popular

And the way it worked was basically

that we provided an installer image

which was sort of a firmware image

as a lot of those NAS devices

have like a firmware upgrade mechanism

and so we would basically fake

we would create a firmware image

which the software would accept

but instead of a firmware update

it would actually contain the Debian installer

so you would install that upgrade

you would reboot, and

you could ssh to the installer

and obviously, in order to ssh to the installer

it needs to bring up the network

and there is a tool called oldsys-preseed

that basically reads the network configuration

from the device

and then sets up the network

and it can always recognize the DHCP

and then the user connects via ssh

again, there would be some indication, maybe

you know, there would be a "beep"

or maybe change the LED

to indicate that the installer is ready

and then, the user basically performs

a regular installation

with normal d-i; they don't have to

do anything differently.

And at the end, flash-kernel runs

to make the system bootable

flash-kernel is called that way

because it used to support
flash-kernel is called that way

because it used to support

like the initial device it supported

booted from flash

but it also generates bootable devices

bootable images of disk devices

so it really, flash-kernel really requires

understanding of each device

and our philosophy

in those cases, with those NAS devices

was really,

we don't touch anything

in the firmware, like

in the configuration,

so sometimes

instead of changing the root device

in the u-boot config we would actually

hard-code that in the ramdisk

just because we wanted people to go back

to the original firmware if they had to

if they had to send it in for repair or something

and

and that kind of approach really worked well

so I think we really had a lot of

people, lot of users running

in Debian in those kind of NAS devices

and it was really easy to do.

You get a firmware image,

you connect to the installer via ssh,

and it just works, it's a normal

debian-installer, the way

everybody knows it,

because some of the other distros

they basically provide, like,

tarballs and instructions

so you need to partition a disk,

need to un-tar it,

you need to change those files,

and even if it sounds simple,

it's so many steps that

you always... you often get something wrong

and then you put the drive into the

NAS device, and it doesn't boot, and you don't know why

where did you make that mistake,

which step

and you basically have to start from scratch

so Debian really provided something unique

by adding

the debian-installer support

but anyway, that's the way

it sort of used to work.

Nowadays, with a lot of those bareboard

it's much easier

I'll get to that.

So at the moment there are three different ARM ports:

There is the old armel

which used to be the "newer" ARM

and now it's old,

and one of the discussions

that we are probably going to have later today on the BoF

is about, should we remove that

after Stretch.

There is armhf and the arm64

and we get basically the question I hope to answer.

If, you know, device A works,

I have a device which is really similar,

but it doesn't work. Why is that?

So, heh,

There have been a lot of changes

in various upstream projects, specially

the kernel and U-boot

that really make things easier.

So, in the past

we basically had

a kernel image

for each platform,

where a platform is basically like

a SSE family

and there would be a different image

because ARM takes a long time

to compile, there was always some debate

about adding a new platform

because there would be a new

image flavor, which takes some time,

and that was just really like,

we couldn't just have one ARM kernel

which works everywhere.

And a lot of people didn't understand

why you had a different kernel

because different platforms,

although there has been a lot of progress upstream,

at least nowadays, with armhf

and with arm64

we just have one kernel.

And upstream basically

Maybe some of you remember the rant

by Niels about

the ARM people doing everything

in different ways,

and there has been a lot of standarization

over the years.

Basically, the other thing is

there used to be for each device

there was a board file

it was like a C file

to initialize the different components,

and the boot loader would

pass the machine ID to the kernel,

and it would load that boot file.

And nowadays,

there is basically a

device tree in the kernel,

which is a description of the hardware

and it will basically compile it

to, like, a binary blob, the .dtb.

And, so, basically you just need

the kernel image, and

the .dtb, which is hardware-specific,

and then it boots.

And obviously, for us in Debian

that makes it much much easier

to support a lot of devices.

The other thing that changed in U-boot,

when you install the Debian kernel image,

it creates

the vmlinux file

and also the RAM disk

but with U-boot,

you couldn't actually load those files

correctly. You basically

had to wrap them in a U-boot image

and it's not really hard

it's just the command initram,

but all of the different devices had

different load addresses,

and, again, that's hardware-specific knowledge

that flash-kernel needed to know.

And nowadays,

there is a command which

directly loads the kernel,

so that, again, was a step to make things easier.

And the last thing which really made

things easier is distro support

in U-boot.

So, basically, in the past

every U-boot, every

devices in U-boot,

booted in a different way,

just in terms on where would it load

the file from, or

what variables would it use, and

nowadays there is something called distro-support

which is basically a standardized way

to boot a Linux distro

with U-boot.

There are basically

two ways, either can

read a config file,

or it can basically run a

boot script, and that's what

we use in Debian, so we basically have

a generic boot script

which loads the kernel,

loads the ramdisk, the .dtb,

and boots Debian.

It can use the generic

bootscript in almost all

of the modern devices.

So nowadays, because of that,

it's much more standard, and

it's much easier to support

those devices.

So here are some examples of flash-kernel

So, basically, flash-kernel has like a database

of devices which it supports

so it's like the machine entry

in /proc/cpuinfo,

and then the kernel flavors

it can run

and, so, this one is

an old device, it still uses

a board file, and it

needs the U-boot wrapper,

So...

You are basically setting the machine ID

those are the flash partitions

where the kernel and the

ramdisk are stored, and

that's the load address for the

U-boot wrapper

And, now, that's a device

which used to have

a boot file,

and which then migrated

to a .dtb,

so basically, heh,

that was really really painful for Debian

so, basically, the kernel people said

"Oh, you are going to move to device tree,

so don't worry,

we are not gonna remove those old board files,

you can still use them."

And then, a few years later they realized

it's really hard to keep both alive,

and they got rid of the board files.

And that was really painful to us

because we had to migrate

So you can see,

here, a kernel version,

and so, from that kernel on,

you needed to use the new way.

And one of the reasons it was painful

specially on the QNAP devices

is because with

they actually have two different CPUs

so there are different barriers

and whilst the board files,

the same board file worked

regardless of the CPU,

because of the way the device tree worked

it actually needed two different device trees

depending on the CPU

so now we basically, you know, something that just worked,

you know, it used to work fine,

you just had one kernel

with the machine ID

at least would work,

and somehow with the device tree

we needed to figure out which one

we need, and so

[?] fortunately did all of that work.

So that's actually a script that runs

to figure out which device tree

that particular device needs.

So now, that's actually

the reason I want to show this is

how simpler things are these days, so

this is an example from

[?] platform, which uses distro-support,

and the only thing you really need is

a machine entry with the name,

and then

like the kernel-flavor,

but that's the same for...

you know, there is only one kernel flavor.

And then, the device tree ID

and, again, all of that stuff is generic,

so it just uses the generic bootscript,

and the generic boot path,

so basically

all it pretty much needs for

a new device is, like,

you know,

these two entries,

it's really really simple.

So here, I just wanted to

tell about the different ARM ports and...

So, for me it was really hard to structure this, because

those changes in the kernel

and U-boot

you know, happen independent of our ARM ports,

But at the same time, because

because the armhf world is much newer,

it works in a different way.

So, basically, the armel

we still have different flavors

but we have already combined the orion and the kirkwood

into one marvell flavor,

and we have the versatile one.

So one of the problems we have in armel is that

a lot of those NAS devices

boot from flash

and they only have, some of them,

3MB for the kernel,

which used to be a lot of space

but nowadays it's not.

So that really puts a lot of restrictions

so basically we disable some stuff

from armel

but I think that armel is really really widely used

because of those NAS devices,

but I think they are slowly getting old,

but there are still a lot of people that use them.

Like I said, it requires the U-boot image

Originally, we used board files

and device tree, most of them

have switched over to device tree

even if some of them still use board files

and adding a new device requires

a number of changes in the installer.

So basically, you needed to map

the device to

the complete image

and there are just there a couple of

things, so basically, any one device

you have to change like five different

places in the installer.

And it was quite confusing for people who wanted

to add new devices

Because it isn't really documented very well

So some examples are a list of three people

who are involved in that port

So, Roger is someone

who is quite new, and who really got

involved into porting those old devices.

And so, armhf is much nicer,

so the majority of

devices support distro-support.

And the other thing we do

is, for some of the devices

we provide SD card images,

which contain

U-boot and the Debian installer

So basically Vagrant

maintains U-boot in Debian, and

a lot of those devices are supported

in U-boot in Debian.

So we just

provide an SD image, so you can just

store it in the SD card, you put it in

and, because of the distro-support,

it just loads debian-installer,

you do the installation, reboot,

and things just work.

So I think it really has gone

...Come a long way.

So nowadays,

adding a new device

requires much fewer changes

than it used to be.

Because as everything uses the same, like, one kernel flavor,

you don't need to patch all those different places anymore,

and because of distro-support, it just works.

You can pretty much use the generic bootscript.

So, arm64.

So, the problem until recently

is that there simply wasn't any hardware

that people could buy.

And that's changing rapidly now

[?]

We have for example the Raspberry Pi 3

which uses a 64 bit CPU

but the software it ships is only 32 bits,

but most of the work

is now

upstreamed to run 64 bit on it

and there are a lot of

devices which sort of work but not quite

but anyway, so the idea for arm64

is that a lot of the

new hardware

specially on the server side

will use UEFI,

and so you basically just

it just works like a PC.

So you have Grub,

you can run debian-installer from Grub

and you get Grub afterwards.

And just boots like a PC,

and Steve has done a lot of work in that area,

And in theory,

it should just work out of the box.

So, if it uses

UEFI, we don't need to add anything.

It just works.

There's nothing to do.

At least, that's the theory.

So, what I found is

...I made a disclaimer, so

...Assuming the kernel

you know, kernel support and stuff...

...To be honest,

right now, that's a big assumption.

So I've been playing with a few

64 bit ARM boards

and basically, what I found is

there is some support upstream,

so I can boot a kernel,

but, oh! There is no USB support!

there is no Wifi support!

So I cannot actually do anything with it.

But I can get to something that

because arm64 is still farily new.

And there is a lot of work going on upstream

to support the various platforms.

So, I think over time that's really going to be much better.

Even though that UEFI idea is there

in reality,

we are going to see different solutions

for arm64.

So we are going to see UEFI, in particular on servers,

but we will also see U-boot.

So a lot of those bare boards

they have U-boot.

And, so, right now

we can use the distro support

and I think that it works pretty well.

But one thing that SuSE has done

so, they just had the same issue,

they want to support all those devices

but they just want to use the same mechanism everywhere,

so they have actually implemented

UEFI on top of U-boot,

so you can basically use U-boot to load Grub

and then you have Grub

so I think that's something we need to figure out;

do we want to stay with distro-support?

do we want to use

UEFI on top of U-boot?

is that something we want to

give users the option?

And then, Fastboot

is something used

in the Android world

and a lot of those, like, I see a lot of

both bare boards, but also

game consoles and stuff, which

which are sort of Android-oriented,

but which can also run normal Linux,

and they will use like Fastboot or something.

And there are tons of other bootloaders out there

But I would definitively say, like, the good ones

are UEFI and U-boot

So, I gave a similar talk a few weeks ago

and turns out, a lot of non-free

firmware, like, "can I

run that stuff, you know,

purely with free software?"

...And...

So, the thing with ARM is that

there are a lot of different platforms

I'm not sure about all of them,

but some of the platforms

I looked at, yes, you

do need some... There is always something propietary.

So in a lot of cases

I see where you have

you use the proprietary first-stage boot loader,

so the Raspberry Pi is an example,

so, I don't actually have a Raspberry Pi, but

the way I understand it is

you basically put some boot files

in an SD card,

and they are proprietary, and

then they load the kernel.

Or in case of [?]

supported in Debian,

the first stage boot loader would basically

be used to run U-boot,

and then we could use U-boot, and

the U-boot support all of it is free software,

but the first stage boot loader isn't.

I have had, there are some people

working on a free replacement for

the Raspberry Pi, though.

With nVidia Tegra,

which is

something I'm working on at the moment

you also have a tiny first stage boot loader,

but then again, you have U-boot, which is free,

and in that case, you also have some firmware

images for the GPU

and for other stuff.

For the Dragon board,

The Dragon board sounded really interesting

because it actually has a

a graphics chip

which can be used with free software,

so that sounded pretty cool,

but again, it has a

proprietary first

stage boot loader,

and then there is a second stage boot loader

which is actually open source

and then there's actually a U-boot of that,

and then you have some

binary blobs which

also need to be installed in flash to work properly

On the Marvell side, I'm

not really aware of anything.

I have never needed to flash anything

proprietary, but

I'm not sure how U-boot gets started

on Marvell, so maybe there is something.

So the future is all... So NAS devices are

like I said, that's something that's

really been popular in Debian.

Specially on the QNAP devices,

but the problem is, so, QNAP...

so the devices we currently support

are pretty old

nowadays,

and they have some newer devices,

but they are not properly supported

in the upstream kernel.

So, I have no plans

to support Debian on those.

I recently did some work on

Seagate NAS devices

which are actually pretty interesting

but again, they will go out of date

already,

and then there's the whole 64 bit ARM

so I think people are really waiting

for arm64 servers

I think there's going to be a whole lot of

work on that area.

And then, there are all of those development boards

Raspberry Pi,

Pine64,

Allwinder,

Basically, all of them

sound exciting, but if you

look at them, all of them

have some upstream issues, so it's...

It is really quite frustrating at the moment.

But I think things are really

moving [?]

And then there's this 96Board initiative,

which is actually done by Linaro,

and, so, Linaro supports

Linux on ARM, so you'd think, "wow,

there are some really nice boards coming out!"

and they differentiated

between the consumer edition and the enterprise edition

beside, I bought some consumer-edition boards,

and it's really horrible.

So first of all, I had to spend like half a day

just putting the components

because it uses, like, a nonstandard power supply,

a nonstandard serial console,

so finally I found

all the pieces I needed, and then

I was expecting, well, it's from Linaro!

Surely everything just works upstream!

But it doesn't. It's like

yes, I can boot the Linux kernel,

but there's no USB, there's no

Wifi, no nothing...

So, yes, that's a little bit frustrating.

On the enterprise edition,

I think that looks more interesting.

You know, that's more standardized.

But again, there have been some delays.

[audience; unintellegible]

Yes, OK.

So that would be interesting to see.

So, the questions that I actually have nowadays

is, so...

Because of those changes,

because of having, you know,

a mode of U-boot, most of those

new devices having a U-boot

with distro-support,

having a kernel which, you know,

one kernel image that works on all of those

devices which are supported,

it's really easy to support

a new device?

As long as it's supported by the kernel.

But, at the same time, I think it's a big

challenge for Debian.

Because right now, if you look at armhf,

we basically say, well,

hint the installer,

and if there's a device tree,

it's probably going to work.

But, what does that mean, right?

And...

which really work, which means,

so, we have Vagrant

having U-boot support

in Debian, we have

people testing debian-installer,

testing the Debian kernel, so we have

devices which really work, well supported,

and then we have some devices

on the other hand where, well, yes,

there is a device tree, but no one

has ever tried it. And

at the moment, we have no way

for users to differentiate

those use case... Those

support levels.

So I'm wondering if we need

like a table,

somewhere where maybe

some support levels, like green, yellow...

red?

Where green is, "yes, we have Debian

people who have

testing that stuff",

yellow would be, "well, we have heard some report

that it might work", and

green is, you know, doesn't work,

or we haven't tried that it works.

Maybe we need something like that.

But right now,

I see from users

"I want to run Debian on my ARM device", and

they don't know if it's going to work.

Yes, then...

[audience; unintellegible]

Yes, so Ben is saying we also need

to track what the earliest and the latest

kernel versions that there have been tested.

So I think we really need to

come up with some criteria

to have, you know, define those kernel support levels that indicate that

And yes, the other question is related

to, with all those boards,

how do we actually test them?

So, I keep --

it's really... remember [?]

which is great, but also acknowledge

those boards, they are so cheap!

so it's like, "oh, there's a new board!

It's £30! I'll just get it!"

And then we have, like, those piles of boards

and realize, well, I don't have time

for testing all of that stuff!

I think that's going to be a real challenge.

hundreds -- I don't know,

it's really hundreds of boards coming out.

How do we support all of that?

So, I think that's the question I wanted to raise,

and maybe something we can talk about in the BoF

But yes, I'm obviously

open for questions now.

[James / purpleidea:] Hello.

So, don't quote me exactly,

so, I believe

RedHat has this problem as well, obviously,

I mean, they are obviously more interested in the

ARM64-only server stuff,

but I believe the way they are going about it

maybe it's something could collaborate on

is, they are trying to make sure and push

all the vendors to be standardized

because, as you pointed out,

it's just not, it's crazy

with all the different device tree differences and so on,

so I believe their strategy is to

work with the vendors, and

require everything to be upstreamed, and

no device tree

specifically, to push everything to just boot

with one kernel, and so on.

So maybe that could be

sacrifice a few of the

shitty boards, but work with the vendors

that make the ones that are upstreamed.

[Martin:] Yes, so I may be mistaken, but

as far as I know, RedHat

basically says, "we only support

UEFI and ACPI"

and that's fair enough,

and I think that works for them, because

it's just the server world they care about,

but I think in the case of Debian

there are so many boards out there which

don't need those specs,

and we sort of live, like, in the "real world"

so I don't think we can

Well -- I know it's different. I mean,

if RedHat wants to target

the server people, like, the people

which give money

that's fair enough, that makes sense for them

but Debian, we run everywhere

so I think we need to support

all those weird

cases as well.

And maybe if it gets too weird,

I mean,

I put in some work in the Dragonboard,

and then I realized,

am I actually spending the time because

no one, like, I have heard

from no one that they have

that board, I mean, like [?]

for the Raspberry Pi, but I have heard

like, pretty much no one has the

Dragonboard, so maybe,

...it's not worth my while,

but if there's a board

that people want to use it,

I think we should support it in Debian.

And we do have the infrastructure

so we, you know, it doesn't have to be

UEFI and ACPI,

we can support other devices, because

we have done it before.

But I definitively agree with

the point about getting more standardization

and that stuff, so

I agree, yes.

And the whole distro-support

in U-boot, that has really made things easier for us.

[Phil:] I may begin to say some of the same things Steve has, anyway.

Steve]: So, yes, on the

ACPI versus DT thing,

it's more a question

of the quality of the implementation of the

of the firmware,

and the upstream support that is...

DT isn't fundamentally

worse for upstream support

than ACPI is,

RedHat are

have some reasons

[?] of sensibility,

but then, it's not

fundamentally, it doesn't fundamentally stop it

people supporting it, specially

a community distro like Debian.

I was also going to ask Martin,

have you talked to

people like kernel CI

about testing and coverage stuff?

[Martin:] Yes, not really, but

I think there are some things we should look into.

[Steve:] Yes, because the whole --

Obviously, making sure the kernel boots

on random hardware

stuff, is exactly what that's doing

and there's at least infrastructure there that might be

nice to play with.

[Martin:] Yes. I think that's a good idea.

[sledge:] So, on the whole 96Boards thing,

sorry!

Most of the engineers involved

are totally aware of

how [?] it really is.

We tried to tell management

they would --

They don't want to listen.

Umm.. So,

there has been a huge amount of pushback

saying, you know, "we're Linaro

we should make sure this is all upstreamed".

[Martin:] Well, I am aware

it's just, Linaro

has a really good brand,

so people expect, you know, when I saw

96Boards it's Linaro,

obviously that stuff it's going to work!

and that's why I was really disappointed.

[Steve:] One positive thing there

is that the first DE board

the SoC does basically work upstream

already with current upstream kernel

it's not like there's some patches still

to go in. Apart from the PCI, it should basically

work... So, hopefully,

hopefully, that's going to be a much

less spectacular

story than the

consumer-edition boards have been.

As they are unfortunate.

[Sledge:] The first EE board

the [?]board, is due to start shipping

really soon now.

There are examples already out there

with engineers for validation.

Literally, we are talking the next couple of weeks

is what I have been told.

That's the first 96Boards I would

actually spend any time on at all personally,

the CE boards are a joke.

I forgot what else I was going to say. Oh!

Yes! Then...

I guess we will go through a lot of this again

in the ARM BoF later on

in terms of

which things we support.

We will carry on with the discussion.

[Vagrant:] Um... Is this working?

Yes, so...

We have got all this great support for

distro-boot command and U-boot, and

now we are starting to get

to UEFI emulation

in U-boot, so even for boards

that don't have UEFI emulation--

UEFI support, if they support

U-boot, it might work

although that means all the standarization

we worked towards, we throw it away.,

Meh!

[Martin:] Yes, I think that's something we need to figure out,

which one do we want to standardize on,

or is that something we want to

leave up to the user.

but yes,

I mean, I think one

I mean, I guess it was

so confusing, because there are like

all those different ways, but

what I really wanted to show is that

things are getting much more

standardized and much better.

So, it used to be really

pretty horrible, and nowadays,

you know, if a platform

uses a modern

U-boot, if it has

upstream support, then it is

actually pretty trivial to add

Debian support, and

I think that wasn't the case

in the past.

I think Ben had another one?

[Ben:] So, I'm

[laughs]

I'm interested to know how much

how much interest there is among the ARM porters

in graphics support.

GPU support, rather

than settling for dumb framebuffers.

How many...

How many of these boards...

Quite a lot of these boards do have HDMI

and I don't know

to what degree they are actually supported.

Are any current projected work

which definitively is using the

GPU on ARM system, and

what's in Debian,

is the Debian base system

does Debian... What's in Debian does not

support GPU.

[Martin:] So, I can't speak for Vagrant...

I have personally done some work

on the Tegra recently,

and nVidia is actually

working upstream on [?]

so I'm really interested in it

but I think that a lot of

the board used, there's money

GPUs I'm not sure would be

...you can get a [laughs]

can we get a microphone for Steve?

[Sledge:] So, on the Mali front,

there has been a lot of haste where we

have been scared of releasing any of this

in any way, shape or form,

without huge

[?]-relays and all that kind of stuff.

So it's not really distributable.

Even, at all.

We are a long way of it being releasable free,

There are packages coming that we

should be getting shippable in non-free

at least if you want to get full acceleration

on your Mali stuff.

Hopefully, that will solve some of the problems

There is a lot, there are a lot of people

in [?] who are very very keen on

supporting it properly, free and upstreamed.

It's a long fight.

It will take a while.

I mean, most of the Mali driver developers

themselves I have spoken to, would love to

make it free. It just comes down

to legal people

being scared.

[Tobi:] Are there some questions left?

[Martin:] Well, thanks for coming!

And there is going to be the ARM BoF

later today, and

Vagrant is also going to talk about his experience

about running a build network

on armhf boards..

Thanks!

[audience clapping]