Software Heritage - Preserving the Free Software Commons

Hi, thank you.

I'm Nicolas Dandrimont and I will indeed
be talking to you about

Software Heritage.

I'm a software engineer for this project.

I've been working on it for 3 years now.

And we'll see what this thing is all about.

[Mic not working]

I guess the batteries are out.

So, let's try that again.

So, we all know, we've been doing
free software for a while,

that software source code is something
special.

Why is that?

As Harold Abelson has said in SICP, his
textbook on programming,

programs are meant to be read by people
and then incidentally for machines to execute.

Basically, what software source code
provides us is a way inside

the mind of the designer of the program.

For instance, you can have,
you can get inside very crazy algorithms

that can do very fast reverse square roots
for 3D, that kind of stuff

Like in the Quake 2 source code.

You can also get inside the algorithms
that are underpinning the internet,

for instance seeing the net queue
algorithm in the Linux kernel.

What we are building as the free software
community is the free software commons.

Basically, the commons is all the cultural
and social and natural resources

that we share and that everyone
has access to.

More specifically, the software commons
is what we are building

with software that is open and that is
available for all to use, to modify,

to execute, to distribute.

We know that those commons are a really
critical part of our commons.

Who's taking care of it?

The software is fragile.

Like all digital information, you can lose
software.

People can decide to shut down hosting
spaces because of business decisions.

People can hack into software hosting
platforms and remove the code maliciously

or just inadvertently.

And, of course, for the obsolete stuff,
there's rot.

If you don't care about the data, then
it rots and it decays and you lose it.

So, where is the archive we go to
when something is lost,

when GitLab goes away, when Github
goes away.

Where do we go?

Finally, there's one last thing that we
noticed, it's that

there's a lot of teams that work on
research on software

and there's no real big infrastructure
for research on code.

There's tons of critical issues around
code: safety, security, verification, proofs.

Nobody's doing this at a very large scale.

If you want to see the stars, you go
the Atacama desert and

you point a telescope at the sky.

Where is the telescope for source code?

That's what Software Heritage wants to be.

What we do is we collect, we preserve
and we share all the software

that is publicly available.

Why do we do that? We do that to
preserve the past, to enhance the present

and to prepare for the future.

What we're building is a base infrastructure
that can be used

for cultural heritage, for industry,
for research and for education purposes.

How do we do it? We do it with an open
approach.

Every single line of code that we write
is free software.

We do it transparently, everything that
we do, we do it in the open,

be that on a mailing list or on
our issue tracker.

And we strive to do it for the very long
haul, so we do it with replication in mind

so that no single entity has full control
over the data that we collect.

And we do it in a non-profit fashion
so that we avoid

business-driven decisions impacting
the project.

So, what do we do concretely?

We do archiving of version control systems.

What does that mean?

It means we archive file contents, so
source code, files.

We archive revisions, which means all the
metadata of the history of the projects,

we try to download it and we put it inside
a common data model that is

shared across all the archive.

We archive releases of the software,
releases that have been tagged

in a version control system as well as
releases that we can find as tarballs

because sometimes… boof, views of
this source code differ.

Of course, we archive where and when
we've seen the data that we've collected.

All of this, we put inside a canonical,
VCS-agnostic, data model.

If you have a Debian package, with its
history, if you have a git repository,

if you have a subversion repository, if
you have a mercurial repository,

it all looks the same and you can work
on it with the same tools.

What we don't do is archive what's around
the software, for instance

the bug tracking systems or the homepages
or the wikis or the mailing lists.

There are some projects that work
in this space, for instance

the internet archive does a lot of
really good work around archiving the web.

Our goal is not to replace them, but to
work with them and be able to do

linking across all the archives that exist.

We can, for instance for the mailing lists
there's the gmane project

that does a lot of archiving of free
software mailing lists.

So our long term vision is to play a part
in a semantic wikipedia of software,

a wikidata of software where we can
hyperlink all the archives that exist

and do stuff in the area.

Quick tour of our infrastructure.

Basically, all the way to the right is
our archive.

Our archive consists of a huge graph
of all the metadata about

the files, the directories, the revisions,
the commits and the releases and

all the projects that are on top
of the graph.

We separate the file storage into an other
object storage because of

the size discrepancy: we have lots and lots
of file contents that we need to store

so we do that outside the database
that is used to store the graph.

Basically, what we archive is a set of
software origins that are

git repositories, mercurial repositories,
etc. etc.

All those origins are loaded on a
regular schedule.

If there is a very active software origin,
we're gonna archive it more often

than stale things that don't get
a lot of updates