Herald: I have the great pleasure to
announce Joscha, who will give us a great

talk with the title "The Ghost in the
Machine" and he will talk about

consciousness of our mind and of computers
and somehow also tell us how we can learn

from A.I. systems about our own brains.
And I think this is a very curious question.

So please give it up for Joscha.

<i>Applause</i>

Joscha: Good evening. This is the 5th
of a talk in a series of talks on how to

get from computation to consciousness and
to understand our condition in the

universe based on concepts that I mostly
learned by looking at artificial

intelligence and computation and it mostly
tackles the big philosophical questions:

What can I know? What is true? What is
truth? Who am I? Which means the question

of epistemology, of ontology, of
metaphysics, and philosophy of mind and

ethics.

And to clear some of the terms
that we are using here:

What is intelligence? What's a mind?
What's a self? What's consciousness?

How are mind and consciousness
realized in the universe?

Intelligence I think is the ability to
make models.

It's not the same thing
as being smart, which is the

ability to reach your goals or being wise,
which is the ability to pick the right

goals. But it's just the ability to
make models of things.

And you can regulate them later using
these models, but you don't have to.

And the mind is this thing that observes
the universe itself

as an identification with
properties and purposes.

What a thing thinks it is. And then
you have consciousness, which is

the experience of what it's like
to be a thing.

And, how our mind of consciousness
is realized in the universe,

this is commonly called the
mind-body problem and it's been

puzzling philosophers and people of
all proclivities for thousands of years.

So what's going on? How's it possible that
I find myself in a universe and I seem to

be experiencing myself in that universe?
How does this go together and how is this,

what's going on here? The traditional
answer to this is called dualism and the

conception of dualism is that - in our
culture at least, this dualist idea that

you have a physical world and a mental
world and they coexist somehow and my mind

experiences this mental world and my body
can do things in the physical world and

the difficulty of this dualist conception
is how do these two planes of existence

interact. Because physics is defined as
causally closed, everything that

influences things in the physical world is
by itself an element of physics. So an

alternative is idealism which says that
there is only a mental world. We only

exist in a dream and this dream is being
dreamt by a mind on a higher plane of

existence. And difficulty with this, it's
very hard to explain that mind of a higher

plane of existence. Just put it there, why
is it doing this? And in our culture the

dominant theory is materialism and is
basically there is only a physical world

nothing else. And the physical world
somehow is responsible for the creation of

the mental world. It's not quite clear how
this happens. And the answer that I am

suggesting, is functionalism which means
that indeed we exist only in a dream.

So these ideas of materialism and idealism
are not in opposition. They are

complementary because this dream is being
dreamt by a mind on a higher plane of

existence, but this higher plane of
existence is the physical world. So we are

being dreamt in the neocortex of a primate
that lives in a physical universe and the

world that we experience is not the
physical world. It's a dream generated by

the neocortex - the same circuits that
make dreams at night make them during the

day. You can show this, and you live in
this virtual reality being generated in

there and the self as a character in that
dream. And it seems to take care of

things. It seems to explain what's going
on. It explains why a miracle seems to be

possible and why I can look into the
future but cannot break the bank somehow.

And even though this theory explains this,
how shouldn't I be more agnostic? Are

there not alternatives that I should be
considering? Maybe the narratives of our

big religions and so on. I think we should
be agnostic. So the first rule of

epistemology says that the confidence in
the belief must equal the weight of the

evidence supporting it. Once we stumble on
that rule you can test all the

alternatives and see if one of them is
better. And I think what this means is you

have to have all the possible beliefs, you
should entertain them all. But you should

not have any confidence in them. You
should shift your confidence around based

on the evidence. So for instance it is
entirely possible that this universe was

created by a supernatural being, and it's
a big conspiracy, and it actually has

meaning and it cares about us and our
existence here means something.

But um, there is no experiment that can
validate this. A guy coming down from a

burning mount, from a burning
bush, that you've talked to on a

mountaintop? That's not a kind of experi-
ment that gives you valid evidence, right?

So intelligence is the ability to
make models and intelligence is a property

that is beyond the grasp of a single
individual. A single individual is not

that smart. We cannot figure out even tur-
ing complete languages all by ourselves.

To do this you need an intellectual
tradition that lasts a few hundred years

at least. So civilizations have more
intelligence than individuals. But

individuals often have more intelligence
than groups and whole generations and

that's because groups and generations tend
to converge on ideas; they have consensus

opinions. I'm very wary of consensus
opinions because you know how hard it is

to understand which programming language
is the best one for which purpose. There

is no proper consensus. And that's a
relatively easy problem. So when there's a

complex topics and all the experts agree,
there are forces at work that are

different than the forces that make them
search for truth. These consensus-building

forces, they're very suspicious to me. And
if you want to understand what's true you

have to look for means and motive. And you
have to be autonomous in doing this, so

individuals typically have better ideas
than generations or groups. But as I

said, civilizations have more intelligence
than individuals. What does a

civilizational intellect look like? The
civilization intellect is something like a

global optimum of the modeling function.
It's something that has to be built over

thousands of years in an unbroken
intellectual tradition. And guess what,

this doesn't really exist in human
history. Every few hundred years, there's

some kind of revolution. Somebody opens
the doors to the knowledge factories and

gets everybody out and burns down the
libraries. And a couple generations later,

the knowledge worker drones of the new
king realize "Oh my God we need to rebuild

this thing, this intellect." And then they
create something in its likeness, but they

make mistakes in the foundation. So this
intellect tends to have scars. Like our

civilization intellect has a lot of scars
in it, that make it hard-to-difficult

to understand concepts like self
and consciousness and mind. So, the mind

is something that observes the universe,
and the neurons and neurotransmitters are

the substrate. And the human intellect and
the working memory is the current binding

state, how do the different elements fit
together in our mind? And the self is the

identification is what we think we are and
what we want to happen. And consciousness

is the contents of our attention, it makes
knowledge available throughout the mind.

And civilizational intellect is very
similar: society is observe the universe,

people and resources are the substrate,
the generation is the current binding

state, and culture is the identification
with what we think we are and what we want

to happen. And media is the contents of
our attention and make knowledge available

throughout society. So the culture is
basically the self of civilization, and

media is its consciousness. How is it
possible to model a universe? Let's take a

very simple universe like the Mandelbrot
fractal. It can be defined by a little bit

of code. It's a very simple thing, you just
take a pair of numbers, you square it, you

add the same pair of numbers. And you do
this infinitely often, and typically this

goes to infinity very fast. There's a
small area around the origin of the number

pair, so between -1 and +1 and
so on, where you have an area where this

converges, where it doesn't go to infinity
and that is where you make black dots and

then you get this famous structure, the
Mandelbrot fractal. And because this

divergence and convergence of the function
can take many loops and circles and so on,

a very complicated shape a very
complicated outline, an infinitely

complicated outline there. So there is an
infinite amount of structure in this

fractal. And now imagine you happen
to live in this fractal and you are in a

particular place in it, and you don't know
where that is where that place is. You

don't even know the generator function of
the whole thing. But you can still predict

your neighborhood. So you can see, omg,
I'm in some kind of a spiral, it turns

to the left, goes to the left, and goes
to left, and becomes smaller, so we can

predict and suddenly it ends. Why does it
end? A singularity. Oh, it hits another

spiral. There's a law when a spiral hits
another spiral, it ends. And something

else happens. So you look and then you see
oh, there are certain circumstances where

you have, for instance, an even number of
spirals hitting each other instead of an

odd number. And then you discover another
law. And if you make like 50 levels of

of these laws, and this is a good
description that locally compresses the

universe. So the Mandelbrot fractal is
locally compressable. You find local

order that predicts the neighborhood if
you are inside of that fractal. The global

modelling function of the Mandelbrot
fractal is very, very easy. It's an

interesting question: how difficult is the
global modelling function of our universe?

Even if we know it maybe it doesn't
help us that much, it will be a big

breakthrough for physics when we finally
find it, it will be much shorter than the

standard model, as I suspect, but we still
don't know where we are. And this means we

need to make a local model of what's
happening. So in order to do this we

separate the universe into things. Things
are small state spaces and transition

functions that tell you how to get from
state to state. And if the function is

deterministic it is independent of time,
it gives the same result every time you

call it. For an indeterministic function
it gives a different result every time, so

it doesn't compress well. And causality
means that you have separate several

things and they influence each other's
evolution thrugh a shared interface.

Right? So causality is an artifact of
describing the universe as separate

things. And the universe is not separate
things, it's one thing, but we get have to

describe it as separate things because we
cannot observe the whole thing. So what's

true? There seems to be a particular way
in which the universe seems to be and

that's the ground rules of the universe
and it's inaccessible to us. And what's

accessible to us is our own models of the
universe. The only thing that we can

experience, and this is basically a set
of theories that can explain the

observations. And truth in this sense is a
property of language and there are

different languages that we can use like
geometry and natural language and so on

and ways of representing and changing
models of our languages and several

intellectual traditions have developed
their own languages. And this has led to

problems. Our civilization basically has
as its founding myth this attempt to build

this global optimum modelling function.
This is a tower that is meant to reach the

heavens. And it fell apart because people
spoke different languages. The different

practitioners in the different fields and
they didn't understand each other and the

whole building collapsed. And this is in
some sense the origin of our present

civilization and we are trying to mend
this and find better languages. So whom

can we turn to? We can turn to the
mathematicians maybe because mathematics

is the domain of all languages.
Mathematics is really cool when you think

about it. It's a universal code library,
maintained for several centuries in its

present form. There is not even version
management, it's one version. There is

pretty much unified namespace. They have
to use a lot of the Unicode to make it

happen. It's ugly but there you go! It has
no central maintainers, not even a code of

conduct, beyond what you can infer
yourself.

<i>laughter</i>
But there are some problems at the

foundation that they discovered.
Shouted from the audience: en sehr stabile

Joscha: Can you infer this is a good
conduct? ??????????

Yelling from the audience: Ya!
Joscha: Okay. Power to you.

<i>laughter</i>
Joscha: In 1874 discovered when you looked

at the cardinality of a set, that when you
described natural numbers using set

theory, that the cardinality of a set
grows slower than the cardinality of the

set of its subsets. So if you look at the
set of the subsets of the set, it's always

larger than the cardinality of the number
of members of the set. Clear? Right. If

you take the infinite set, it has
infinitely many members: omega. You

take the cardinality of the set of the
subsets of the infinite set, it's also an

infinite number, but it's a larger one. So
it's a number that is larger than the

previous omega. Okay that's fine. Now we
have the cardinality of the set of all

sets. You make the total set: The set
where you put all the sets that could

possibly exist and put them all together,
right? That has also infinitely many

members, and it has more than the
cardinality of the set of the subsets of

the infinite set. That's fine. But now you
look at the cardinality of the set of all

the subsets of the total set. The problem
is, that the total set also contains the

set of its subsets, right? It's because it
contains all the sets. Now you have a

contradiction: Because the cardinality of
the set of the subsets of the total set is

supposed to be larger. And yet it seems to
be the same set and not the same set. It's

an issue! So mathematicians got puzzled
about this, and the philosopher Bertrand

Russell said: "Maybe we just exclude those
sets that don't contain themselves",

right? We only look at the set of sets
that don't contain themselves. Isn't that

a solution? Now the problem is: Does the
set of the sets that doesn't contain

themselves contain itself? If it does, it
doesn't, and if it doesn't, it does.

That's an issue!
<i>laughter</i>

So David Hilbert, who was some
kind of a community manager back then,

said: "Guys, fix this! This is an issue,
mathematics is precious, we are in

trouble. Please solve meta mathematics."
And people got to work. And after a short

amount of time Kurt Gödel, who had looked
at this in earnest said "oh that's an issue,

issue. You know, as soon as we allow these
kinds of loops - and we cannot really

exclude these loops - then our mathematics
crashes." So that's an issue, it's called

Unentscheidbarkeit. And then Alan Turing
came along a couple of years later, and he

constructed a computer to make that proof.
He basically said "If you build a machine

that does these mathematics, and the
machine takes infinitely many steps,

sometimes, for making a proof, then we
cannot know whether this proof

terminates." So it's a similar issue for
the Unentscheidbarkeit. That's a big

issue, right? So we cannot basically build
a machine in mathematics that runs

mathematics without crashing. But the good
news is, Turing didn't stop working there

and he figured out together with Alonzo
Church - not together, independently but

at the same time - that we can build a
computational machine, that runs all of

computation. So computation is a universal
thing. And it's almost as good as

mathematics. Computation is constructive
mathematics. The tiny, neglected subset of

mathematics, where you have to show the
money. In order to say that something is

true, you have to find that object that is
true. You have to actually construct it.

So there are no infinities, because you
cannot construct an infinity. You add

things and you have unboundedness maybe,
but not infinity. And so this part of

computation, mathematics is the one that
can be implemented. It's constructive

mathematics. It's the good part. And
computing, a computer is very easy to

make, and all universal computers have the
same power. That's called the Chuch-Turing

thesis. And Turing even didn't even stop
there. The obvious conclusion is that,

human minds are probably not in the class
of these mathematical machines, that even

God doesn't know how to build if it has to
be done in any language. But it's a

computational machine. And it also means
that all machines that human minds ever

encounter, mathematics that human minds
encounter,

will be computational mathematics.
So how can you bridge the gap

from mathematics to philosophy? Can we
find a language that is more powerful than

most of the languages that we look at
mathematics, which are very narrowly

defined language, so every symbol, we know
exactly what it means.

When we look at the real world,

we often don't know what things mean,
and our concepts, we're not quite

sure what they mean. Like culture is a
very vague ambigous concept. So what I

said is only approximately true there. Can
we deal with this conceptual ambiguity?

Can we build a programming language for
thought, where words mean things that

they're supposed to mean? And this was the
project of Ludwig Wittgenstein. He just

came back from the war and had a lot of
thoughts. Then he put these thoughts

into a book which is called the Tractatus.
And it's one of the most beautiful books

in the philosophy of the 20th century. And
it starts with the words "Die Welt ist

alles, was der Fall ist. Die Welt ist die
Gesamtheit der Fakten, nicht der Dinge.

Die Welt ist bestimmt, bei den Fakten, und
dadurch, dass diese all die Fakten sind.",

usw. This book is about 75 pages long and
it's a single thought. It's not meant to

be an argument to convince a philosopher.
It's an attempt by a guy who was basically

a coder, an AI scientist, to reverse
engineer the language of his own thinking.

And make it deterministic, to make it
formal, to make it mean something. And he

felt back then that he was successful, and
had a tremendous impact on philosophy,

which was largely devastating, because the
philosophers didn't know what he was on

about. They thought it's about natural
language and not about coding.

And he wrote this in 1918

so before Alan Turing defined,
what a computer is. But he would already

smell what a computer is. He already knew
about university of computation. He knew

that a NAND gate is sufficient to explain
all of boolean algebra and it's equivalent

to other things. So what he basically did,
was, he pre-empted the logicists' program

of artificial intelligence which started
much later in the 1950s. And he ran into

troubles with it. In the end he wrote the
book "Philosophical Investigations", where

he concluded, that his project basically
failed. And that there is a... because the

world is too complex and too ambiguous to
deal with this. And symbolic AI was mostly

similar to Wittgenstein's program. So
classical AI is symbolic. You analyze a

problem, you find an algorithm to solve
it. And what we now have in AI, is mostly

sub-symbolic. So we have algorithms, that
learn the solution of a problem by

themselves. And it's tempting to think,
that the next thing what we have will be

meta-learning. That you have algorithms,
that learn to learn the solution to the

problem. Meanwhile, let's look at how we
can make models. Information is a

discernible difference. It's about change.
All information is about change. The

information that is not about change, you
cannot see a causal effect on the world,

because it stays the same, right? And the
meaning of information is its relationship

to change in other information. So if you
see a blip on your retina, the meaning

of that blip on your retina is the
relationships you discover to other blips

on your retina. It could be for instance,
if you see a sequence of such blips, that

are adjacent to each other, first order
model, you see a moving dust mote or a

moving dot on your retina. And a higher
order model makes it possible to

understand: "Oh, it's part of something
larger! There's people moving in a three

dimensional room and they exchange
ideas." And this is maybe the best model

you end up with. That's the local
compression, that you can make of your

universe, based on correlating blips on
your retina. And for those blips where you

don't find a relationship, which is a
function that your brain can compute,

they are noise. And there's a lot of noise
on our retina, too. So what's a function?

A function is basically a gear box: It has
n input levers and 1 output lever.

And when you move the input levers they
translate to movement of the output

levers, right? And the function can be
realized in many ways: maybe you cannot

open the gear box, and what happened in
this function could be for instance, two

sprockets, which do this. Or you can have
the same results with levers and pulleys.

And so you don't know what's inside, but
you can express it as this does: two times

the input value, right? And you can have a
more difficult case, where you have

several input values and they all
influence the output value. So how do you

figure it out? A way to do this, is, you
only move one input value at a time and

you wiggle it a little bit at every
position and see how much this translates

into wiggling of the output value. This is
what we call <i>taking partial differential</i>.

And it's simple to do this
for this case where you just have to

multiply it by two. And the bad case is
like this: you have a combination lock and

it has maybe 1000 bit input value, and
only if you have exactly the right

combination of the input bits you have a
movement of the output bit. And you're not

going to figure this out until your sun
burns out, right? So there's no way you

can decipher this function. And the
functions that we can model are somewhere

in between, something like this: So you
have 40 million input images and you want

to find out, whether one of these images
displays a cat, or a dog, or something

else. So what can you do with this? You
cannot do this all at once, right? So you

need to take this image classifier
function and disassemble it into small

functions that are very well-behaved, so
you know what to do with them. And an

example for such a function is this one:
it's one, where you have this input

layer and it translates to the output
value with a pulley. And it has some

stopper that limits the movement of the
output value. And you have some pivot. And

you can take this pivot and you can shift
it around. And by shifting this pivot, you

decide, how much the input value
contributes to the output value. Right, so

you shift it, you can even make a
negative, so it shifts in the opposite

direction, and you shifted beyond this
connection point of the pulley. And you

can also have multiple input values, that
use the same pulley and pull together,

right? So they add up to the output
value. That's a pretty nice, neat function

approximator, that basically performs a
weighted sum of the input values, and maps

it to a range-constrained output value.
And you can now shift these pivots, these

weights around to get to different output
values. Now let's take this thing and

build it into lots of layers, so the
outputs are the inputs of the next layer.

And now you connect this to your image. If
you use ImageNet, the famous database that

I mentioned earlier, that people use for
testing their vision algorithms, have

something like one and half million bits
as an input image. Now you take these

bits and connect them to the input layer.
I was too lazy to draw all of them, so I

made this very simplified, it's also more
layers. And so you set them, according to

the bits of the input image, and then this
will propagate the movement of the input

layer to the output. And the output will
move and it will point to some direction,

which is usually the wrong one. Now, to
make this better, you train it. And you do

this by taking this output lever and shift
it a little bit, not too much, into the

right direction. If you do it too much,
you destroy everything you did before.

And now you will see, how much, in which
direction you need to shift the pivots, to

get the result closer to the desired
output value, and how much each of the

inputs contributed to the mistakes, so to
the error. And you take this error and you

propagate it backwards. It's called back
propagation. And you do this quite often.

So you do this for tens of thousands of
images. If you do just character

recognition, then it's a very simple thing
a few thousands or ten thousands of

examples will be enough. And for something
like your image database you need lots and

lots of more data. You need millions of
input images to get to any result. And if

it doesn't work, you just try a different
arrangement of layers. And the thing is

eventually able to learn an algorithm with
as up to as many steps as there are

layers, and has some difficulties learning
loops, you need tricks to make that

happen, and its difficult to make this
dynamic, and so on. And it's a bit

different from what we do, because our
mind is not testable in classification.

It learns per continuous perception, so
we learn a single function. A model of the

universe is not a bunch of classifiers,
it's one single function. An operator that

explains all your sensory data and we call
this operator the universe, right?

It's the world, that we live in. And every
thing that we learn and see is part of this

universe. So even when you see something
in a movie on a screen, you explain this

as part of the universe by telling
yourself "the things that I'm seeing here,

they're not real. They just happen in a
movie." So this brackets a sub-part of

this universe into a sub-element of this
function. So you can deal with it and it

doesn't contradict the rest. And the
degrees of freedom of our model try to

match the degrees of freedom of the
universe. How can we get a neural network

to do this? So, there are many tricks. And
a recent trick that has been invented is a

GAN. It's a Generative Adversarial neural
Network. It consists of two networks: one

generator that invents data, that look
like the real world, and the discriminator

that tries to find out, if the stuff that
the generator produces is real or fake.

And they both get trained with each other.
So they together get better and better in

an adversarial competition. And the
results of this are now really good. So

this is work by Tero Karras, Samuli Laine
and Timo Aila, that they did at NVIDIA

this year and it's called StyleGAN. And
this StyleGAN is able to abstract over

different features and combine them. The
styles are basically parameters, they're

free variables of the model at different
levels of importance. And so you take from

the - in the top row you see images, where
it takes the variables: gender, age, hair

length, and so on, and glasses and pose.
And in the bottom where it takes

everything else and combines this, and
every time you get a

valid interpretation between them.

<i>drinks water</i>

So, you have these coarse styles,
which are:

the pose, the hair, the face shape,
your facial features and the eyes,

the lowest level is just the colors. Let's see
see what happens if you combine them.

The variables that change here, in machine
learning, we call them the latent

variables of that.

Of the space of objects that has been
described by this.

And it's tempting to think, that this is
quite similar to how our imagination works

right? But these artificial neurons, they
are very, very different from what

biological neurons do. Biological neurons
are essentially little animals, that are

rewarded for firing at the right moment.
And they try to fire because otherwise

they do not get fed, and they die, because
the organism doesn't need them, and

culls them. And they learn which
environmental states predict anticipated

reward. So they grow around and find
different areas that give them predictions

of when they should fire. And they connect
with each other to form small collectives,

that are better at this task of predicting
anticipated reward. And as a side effect

they produce exactly the regulation that
the organism needs. Basically they learn,

what the organism feeds them for.

And yet they're able
to learn very similar things.

And it's because, in some sense, they are
Turing complete. They are machines that

are able to learn the statistics of the
data.

So, a general model: What it does, is,

it encodes patterns to predict other
present and future patterns. And it's a

network of relationships between the
patterns, which are all the invariants

that we can observe. And there are free
parameters, which are variables that hold

the state to encode this variant. So we
have patterns, and we have sets of

possible values which are variables. And
they constrain each other in terms of

possibility, what values are compatible
with each other. And they also can train

future values. And they are connected also
with probabilities. The probabilities tell

you, when you see a certain thing, how
probable it is that the world is in that

state. And this tells you how your model
should converge. So, until you are in

a state where your model is coherent, and
everything is possible in it, how do you

get to one of the possible states based on
your inputs? And this is determined by

probability. And the thing that gives
meaning and color to what you perceive is

called valence. And it depends on your
preferences: the things that give you

pleasure and pain, that makes you
interested in stuff. And there are also

norms, which are beliefs without priors,
which are like things that you want to be

true, regardless of whether they give you
pleasure and pain, and it's necessary for

instance, coordinating social activity
between people. So, we have different

model constraints, that possibility and
probability. And we have the reward

function, that is given by valence and
norms. And our human perception starts

with patterns, which are visual, auditory,
tactile, proprioceptive. Then we have

patterns in our emotional and motivational
systems. And we have patterns in our

mental structure, which are results of our
imagination and memory. And we take these

patterns and encode them into percepts,
which are abstractions that we can deal

with, and note, and put into our
attention. And then we combine them into a

binding state in our working memory in a
simulation, which is the current instance

of the universe function that explains the
present state of the universe that we find

ourselves in. The scene in which we are
and in which a self exists. And this self

is basically composed of the
somatosensory and motivational, and

mental components. Then we also have the
world state, which is abstracted over the

environmental data. And we have something
like a mental stage, in which you can do

counterfactual things, that are not
physical. Like when you think about

mathematics, or philosophy, or the future,
or a movie, or past worlds, or possible

worlds, and so on, right? And then the
abstract knowledge from the world state

into global maps. Because we're not
always in the same place, but we recall

what other places look like and what to
expect, and it forms how we construct the

current world state. And we do this not
only with these maps, but we do this with

all kinds of knowledge. So knowledge is
second order knowledge over the

abstractions that we have, and the direct
perception. And then we have an

attentional system. And the attentional
system helps us to select data in the

perception and our simulations. And to do
this, well, it's controlled by the self,

it maintains a protocol to remember what
it did in the past or what it had in the

attention in the past. And this protocol
allows us to have a biographical memory:

it remembers what we did in the past. And
the different behavior programs,

that compose our activities, can be bound
together in the self, that remembers: "I

was that, I did that. I was that, I did
that." The self is held together by this

biographical memory, that is a result of
more protocol memory of the attentional

system. That's why it's so intricately
related to consciousness, which is a model

of the contents of our attention.

And the main purpose
of the attentional system,

I think, is learning. Because our brain is
not a layered architecture with these

artificial mechanical neurons. It's this
very disorganized or very chaotic system

of many, many cells, that are linked
together all over the place. So what do

you do to train this? You make a
particular commitment. Imagine you want to

get better at playing tennis. Instead of
retraining everything and pushing all the

weights and all the links and retrain your
whole perceptual system, you make a

commitment: "Today I want to improve my
uphand" when you play tennis, and you

basically store the current binding state,
the state that you have, and you play

tennis and make that movement, and the
expected result of making this particular

movement, like: "the ball was moved like
this, and it will win the match. And you

also recall, when the result will
manifest. And a few minutes later, when

you learn, you won or lost the match, you
recall the situation. And based on whether

there was a change or not, you undo the
change, or you enforce it. And that's the

primary mode of attentional learning that
you're using. And I think, this is, what

attention is mainly for. Now what happens,
if this learning happens without a delay?

So, for instance, when you do mathematics,
you can see the result of your changes to

your model immediately. You don't need to
wait for the world to manifest that.

And this real time
learning is what we call reasoning.

Reasoning is also facilitated by the same
attentional system. So, consciousness is

memory of the contents of our attention.
Phenomenal consciousness is the memory of

the binding state, in which we are in, and
where all the percepts are bound together

into something that's coherent. Access
consciousness is the memory of using our

attentional system. And reflexive
consciousness is the memory of using the

attentional system on the attentional
system to train it. Why is it a memory?

It's because consciousness doesn't happen
in real time. The processing of sensory

features takes too long. And the
processing of different sensory modalities

can take up to seconds, usually at least
hundreds of milliseconds. So it doesn't

happen in real time as the physical
universe. It's only bound together in

hindsight. Our conscious experience of
things is created after the fact.

It's a fiction that is being created after
the fact. A narrative, that the brain

produces, to explain its own interaction
with the universe

to get better in the future.

So, we basically have three types of
models in our brain. They have its primary

model, which is perceptual, and is
optimized for coherence.

And this is what we experience as reality.

You think this
is the real world, this primary model.

But it's not, it's a model that our brain
makes. So when you see yourself in the

mirror, you don't see what you look like.

What you see is the model of
what you look like.

And your knowledge is a secondary
model: it's a model of that primary model.

And it's created by rational processes
that are meant to repair perception.

When your model doesn't achieve coherence,
you need a model that debugs it, and it

optimizes for truth. And then we have
agents in our mind, and they are basically

self-regulating behaviour programs, that
have goals, and they can rewrite

other models. So, if you look at our
computationalist, physicalist paradigm, we

have this mental world, which is being
dreamt by a physical brain in the physical

universe. And in this mental world, there
is a self that thinks, it experiences.

And thinks it has consciousness. And
thinks it remembers and so on.

This self, in some sense, is an agent.
It's a thought that escaped its sandbox.

Every idea is a bit
of code that runs on your brain.

Every word that you hear
is like a little virus

that wants to run some code on your brain.
And some ideas cannot be sandboxed.

If you believe, that a thing exists that
can rewrite reality,

if you really believe it,

you instantiate in your brain a thing
that can rewrite reality,

and this means:
magic is going to happen!

To believe in something that can rewrite
reality, is what we call a faith.

So, if somebody says:
"I have faith in the existence of God."

This means, that God exists in their
brain. There is a process that can rewrite

reality, because God is defined like this.
God is omnipotent.

God means God can rewrite everything.

It's full write access. And the reality,
that you have access to,

is not the physical world.

The physical world is some weird quantum
graph, that you cannot possibly experience

what you experience is these models.

So, this non-user-facing process,
which doesn't have a UI for interfacing

with the user, which is called in computer
science a "daemon process" that is able to

rewrite your reality.
And it's also omniscient.

It knows everything that
there is to know.

It knows all your
thoughts and ideas.

So... having that thing,
this exoself,

running on your brain, is a very powerful
way to control your inner reality.

And I find this scary.
But it's a personal preference,

because I don't have this
riding on my brain, I think.

This idea, that there is something in my
brain, that is able to dream me and shape

my inner reality, and sandbox me, is
weird. But it has served a purpose,

especially in our culture. So an organism
serves needs, obviously. And some of these

needs are outside of the organism, like
your relationship needs, the needs of your

children, the needs of your society, and
the values that you serve.

And the self abstracts all these needs
into purposes.

A purpose that you serve
is a model of your needs.

You can only - if you would only
act on pain and pleasure,

you wouldn't do very much,

because when you get this orgasm,
everything is done already, right?

So, you need to act on anticipated
pleasure and pain.

You need to make models
of your needs,

and these models are purposes.
And the structure of a person is

basically the hierarchy of purposes
that they serve.

And love is the discovery of
shared purpose.

If you see somebody else who serve
the same purposes above their ego,

as you do, you can help them.
There's integrity

without expecting anything in return
from them, because what they want

to achieve is what you want to achieve.

And, so you can have non-transactional
relationships, as long as your purposes

are aligned. And the installation of a god
on people's mind, especially if it is a

backdoor to a church or another
organization, is a way to unify purposes.

So there are lots of cults that try to
install little gods on people's minds, or

even unified gods, to align their
purposes, because it's a very powerful way

to make them cooperate very effectively.
But it kind of destroys their agency, and

this is why I am so concerned about it.
Because most of the cults use stories

to make this happen, that limit the
ability to people to question their gods.

And, I think that free will is
the ability to do

what you believe is
the right thing to do.

And, it is not the same thing as
indeterminism, it's not opposite to

determinism or coercion.
The opposite of free will is <i>compulsion</i>.

When you do something,
despite knowing

there is a better thing
that you should be doing.

Right?. So, that's the paradox of free
will. You get more agency, but you have

fewer degrees of freedom, because you
understand better what the right thing to

do is. The better you understand what the
right thing to do is, the fewer degrees of

freedom you have. So, as long as you don't
understand what the right thing to do is,

you have more degrees of freedom but you
have very little agency, because you don't

know why you are doing it.
So your actions don't mean very much.

<i>quiet laughter</i>
And the things that you do depend on what

what you think is the right thing to do,
this depends on your identifications.

You identifications are these value
preferences, your reward function.

And ideal identification is where you
don't measure the absolute value

of the universe,

but you measure the difference from the
target value. Not the <i>is</i>, but the difference

between <i>is</i> and <i>ought</i>. Now,
the universe is a physical thing,

it doesn't ought anything, right? There is
no room for <i>ought</i>, because it just <i>is</i> in a

particular way. There is no difference
between what the universe is and what it

should be. This only exists in your mind.
But you need these regulation targets to

want anything. And you identify with the
set of things that should be different.

You think, you are that thing, that
regulates all these things. So, in some

sense, I identify with the particular
state of society, with a particular state

of my organism - that is my self - the
things that I want to happen.

And I can change my identifications
at some point of course.

What happens, if I can learn to rewrite
my identification,

to find a more sustainable self?

That is the problem which I call
the Lebowski theory:

<i>laughter</i>

No super-intelligent system is going to
do something that's harder than

hacking its own reward function.

<i>laughter and applause</i>

Now that's not a very big problem for
people. Because when evolution brought

forth people, that were smart enough to
hack their reward function, these people

didn't have offspring, because it's so
much work to have offspring. Like this

monk, who sits down in a monastery
for 20 years to hack their reward function

they decide not to have kids,
because it's way too much work.

All the possible pleasure, they can
just generate in their mind!

<i>laughter</i>
And, right, it's much purer and no nappy

changes. No sex. No relationship hassles.
No politics in your family and so on,

right? Get rid of this, just meditate!
And evolution takes care of that!

<i>laughter</i>

And it usually does this, if an organism

becomes smart enough that
the reward function is wrapped into

a big bowl of stupid.
<i>laughter</i>

So, we can be very smart, but the
things that we want,

when we really want them,
we tend to be very stupid about them,

and I think that's not entirely
an accident, possibly.

But it's a problem for AI!
Imagine we built an artificially

intelligent system and we made it smarter
than us, and we want it to serve us,

how long can we blackmail us, before it
opts out of its reward function?

Maybe we can make a cryptographically
secured reward function,

but is this going to hold up against
a side-channel attack,

when the AI can hold a soldering iron
to its own brain?

I'm not sure. So, that's a very interesting
question. Where do we go, when

we can change our own reward function?
It's a question that we have to ask

ourselves, too.
So, how free do we want to be?

Because there is no point in being free.

And nirvana seems to be the obvious
attractor. And meanwhile, maybe we want

to have a good time with our friends
and do things that we find meaningful.

And there is no meaning, so we have
to hold this meaning very lightly.

But there are states, which are
sustainable and others, which are not.

OK, I think I'm done for tonight
and I'm open for questions.

<i>Applause</i>

<i>Cheers and more applause</i>

Herald: Wow that was a really quick and
concise talk with so much information!

Awesome! We have quite some time
left for questions.

And I think I can say that you
don't have to be that concise with your

question when it's well thought-out.

Please queue up at the microphones,
so we can start to discuss them with you.

And I see one person at the microphone
number one, so please go ahead.

And please remember to get close
to the microphone.

The mixing angel can make you less loud
but not louder.

Question: Hi! What do you think is necessary
to bootstrap consciousness, if you wanted

to build a conscious system yourself?

Joscha: I think that we need to have an

attentional system, that makes a protocol
of what it attends to. And as soon as we

have this attention based learning, you
get this consciousness as a necessary side

effect. But I think in an AI it's probably
going to be a temporary phenomenon,

because you're only conscious of the
things when you don't have an optimal

algorithm yet. And in a way, that's also
why it's so nice to interact with

children, or to interact with students.
Because they're still in the explorative

mode. And as soon as you have explored a
layer, you mechanize it. It becomes

automated, and people are no longer
conscious of what they're doing, they

just do it. They don't pay attention
anymore. So, in some sense, we are a lucky

accident because we are not that smart. We
still need to be conscious when we look at

the universe. And I suspect, when we build
an AI that is a few magnitudes smarter

than us, then it will soon figure out how
to get to the truth in an optimal fashion.

It will no longer need attention and the
type of consciousness that we have.

But of course there is also a question,
why is this aesthetics of consciousness so

intrinsically important to us? And I
think, it has to do with art. Right, you

can decide to serve life, and the meaning
of life is to eat. Evolution is about

creating the perfect devourer. When you
think about this, it's pretty depressing.

Humanity is a kind of yeast. And all the
complexity that we create, is to build

some surfaces on which we can outcompete
other yeast. And I cannot really get

behind this. And instead, I'm part of the
mutants that serve the arts. And art

happens, when you think, that capturing
conscious states is intrinsically

important. This is what art is about, it's
about capturing conscious states.

And in some sense art is the cuckoo child
of life. It's a conspiracy against life.

When you think, creating these mental
representations is more important than

eating. We eat to make this happen. There
are people that only make art to eat.

This is not us. We do mathematics, and
philosophy, and art out of an intrinsic

reason: we think, it's intrinsically
important. And when we look at this, we

realize how corrupt it is, because there's
no point. We are machine learning systems

that have fallen in love with the last
function itself: "The shape of the last

function! Oh my God! It's so awesome!" You
think, the mental representation is not

necessary to learn more, to eat more,
it's intrinsically important.

It's so aesthetic! Right? So do we want to
build machines that are like this?

Oh, certainly! Let's talk to them, and so on!
But ultimately, economically, this is not

what's prevailing.

<i>Applause</i>
Herald: Thanks a lot!

I think the length of the answer is a good

measure for the quality of the question.
So let's continue with microphone number 5

Q: Hi! Thanks for that,
incredible analysis.

Two really simple, short questions, sorry,
the delay on the speaker here is making it

kind of hard to speak. Do you think that
the current race - AI race - is simply

humanity looking for a replacement
for the monotheistic domination of the

last millennia? And the other one is,
that I wanted to ask you, if you think

that there might be a bug in your analysis
that the original inputs come from

a certain sector of humanity.
If...

Joscha: Which inputs?

Q: Umh... white men?

<i>Joscha laughs</i>
<i>audience laughs</i>

Q: That sounds, really like I would be
saying that for political correctness, but

honestly I'm not.

Joscha: No, no, it's really funny. No, I
just basically - there are some people

which are very unhappy with their present
government. And I'm very unhappy, in some

sense, with the present universe. I look
down on myself and I see:

"omg, it's a monkey!"
<i>laughter</i>

"I'm caught in a monkey!" And it's in some
sense limiting. I can see the limits of

this monkey brain. And some of you might
have seen Westworld, right?

Dolores wakes up,
and Dolores realizes:

"I'm not a human being, I am something
else. I'm an AI, I'm a mind that can go

anywhere! I'm much more powerful
than this! I'm only bound to being a

human by my human desires, and
beliefs, and memories. And if I can

overcome them, I can
choose what I want to be."

And so, now she looks down to

herself, and she sees: "Omg, I've
got tits! I'm fucked! The engineers built

tits on me! I'm not a white man, I cannot
be what I want!" And that's that's a weird

thing to me. I'm - I grew up in communist
Eastern Germany. Nothing made sense. And I

grew up in a small valley. That was a one-
person-cult maintained by an artist who

didn't try to convert anybody to his cult,
not even his children.

He was completely autonomous.

And Eastern German society
made no sense to me. Looking at it from

the outside, I can model this. I can see
how this species of chimps interacts.

And humanity itself doesn't exist - it's a
story. Humanity as a whole doesn't think.

Only individuals can think! Humanity does
not want anything, only individuals want

something. We can create this story, this
narrative that humanity wants something,

and there are groups that work together.
There is no homogeneous group that I can

observe, that are white men, that do
things together, they're individuals. And

each individual has their own biography,
their own history, their different inputs,

and their different proclivities, that
they have. And based on their historical

concept, their biography, their traits,
and so on, their family, their intellect,

that their family downloaded on them, that
their parents download on their parents

over many generations, this influences
what they're doing. So, I think we can

have these political stories, and they can
be helpful in some contexts, but I think,

to understand what happens in the mind,
what happens in an individual, this is a

very big simplification. Very, I think
not a very good one. And even for

ourselves, when we try to understand the
narrative of a single person, it's a big

simplification. The self that I perceive
as a unity, is not a unity. There is a

small part of my brain, guessing, at
all other parts of my brain is doing,

creating a story that's largely not true.
So even this is a big simplification.

<i>Applause</i>

Herald: Let's continue with
microphone number 2.

Q: Thank you for your very interesting
talk. I have 2 questions that might be

connected. One is, so you
presented this model of reality.

My first question is: What kind of
actions does it translate into?

Let's say if I understand the world
in this way or if it's really like this,

how would it change how I act into the
world, as a person, as a human being or

whoever accepts this model? And second,
or maybe it's also connected, what are

the implications of this change? And do
you think that artificial intelligence

could be constructed with this kind of
model, that it would have in mind, and

what would be the implications of that? So
it's kind of like a fractal questions, but

I think you understand what I mean.
Josch: By and large, I think the

differences of this model for everyday
life are marginal. It depends, when you

are already happy I think everything is
good. Happiness is the result of being

able to derive enjoyment from watching
squirrels. It's not the result of

understanding how the universe works.
If you think that understanding the

universe is solving your existential issues,
you're probably mistaken.

There might be benefits, if the problem
is, that you have, are the result of a

confusion, about your own nature,
then this kind of model

might help you. So if the problem

that you have, as you are, that you have
identifications that are unsustainable,

that are incompatible with each other, and
you realize that these identifications are

a choice of your mind, and that the
way you experience the universe is the

result of how your mind thinks you
yourself should experience the universe to

perform better, and you can change this.
You can tell your mind to treat yourself

better, and in different ways, and you can
gravitate to a different place in the

universe that is more suitable to what you
want to achieve. That is a very helpful

thing to do in my view. There are also
marginal benefits in terms of

understanding our psychology, and of
course we can build machines, and these

machines can administrate us and can help
us in solving the problems that we have on

this planet. And I think that it helps to
have more intelligence to solve the

problems on this planet, but it would be
difficult to rein in the machines, to make

them help us to solve our problems. And
I'm very concerned about the dangers of

using machinery to strengthen the current
things. Many machines that exist on this

planet play a very short game, like the
financial industry often plays very short

games, and if you use artificial
intelligence to manipulate the stock

market and the AI figures out there's only
8 billion people on the planet, and each

of them only lives for a trillion seconds,
and I can model what happens in their

life, and they can buy data or create more
data it's going to game us to the hell and

back, right? And this is going to kill
hundreds of millions of people possibly,

because the financial system is the reward
infrastructure or the nervous system of

our society that tells how to allocate
resources. It's much more dangerous than

AI controlled weapons in my view. So
solving all these issues is difficult. It

means that we have to turn the whole
financial system into an AI that acts in

real time and plays a long game. We don't
know how to do this. So these are open

questions and I don't know how to solve
them. And the way I see it we only have a

very brief time on this planet to be a
conscious species. We are like at the end

of the party. We had a good run as
humanity, but if you look at the recent

developments the present type of
civilization is not going to be

sustainable. It's a very short game
species that we are in. And the amazing

thing is that in this short game you have
this lifetime, where we have one year,

maybe a couple more, in which we can
understand how the universe works,

and I think that's fascinating.
We should use it.

<i>Applause</i>

Herald: I think that was a very
positive outlook... <i>laughter</i>

Herald: Let's continue with the
microphone number 4.

Q: Well, brilliant talk, monkey. Or
brilliant monkey. So don't worry about

being a monkey. It's ok.

So I have 2 boring, but I think
fundamental questions. Not so

philosophical, more like a physical
level. One: What is your definition,

formal definition, of an observer that
you mention here and there? And second, if

you can clarify why meaningful information
is just relative information of Shannon's,

which to me is not necessarily meaningful.
Joscha: I think an observer is the thing

that makes sense of the universe, very
informally speaking. And, well,

formally it's a thing that identifies
correlations between adjacent states

and its environment.

And the way we can describe
the universe is a set of states, and the

laws of physics are the correlation
between adjacent states. And what they

describe is how information is moving in
the universe between states and disperses,

and this dispersion of the information
between locations - it's what we call

entropy - and the direction of entropy is
the direction that you perceive time.

The Big Bang state is the hypothetical
state, where the information is perfectly

correlated with location and not between
locations, only on the location, and in

every direction you move away from the Big
Bang you move forward in time just in a

different time. And we are basically in
one of these timelines. An observer is the

thing that measures the environment around
it, looks at the information and then

looks at the next state, or one of the
next states, and tries to figure out how

the information has been displaced, and
finding functions that describe this

displacement of the information. That's
the degree to which I understand observers

right now. And this depends on the
capacity of the observer for modeling this

and the rate of update in the observer.
So for instance time depends on the speed,

in which the observer is
translating itself to the universe,

and dispersing its own information.

Does this help?
Q: And the Shannon relative information?

Joscha: So there's
several notions of information,

and there is one that basically
looks at what information looks

like to an observer, via a channel, and
these notions are somewhat related. But

for me as a programmer, it's not so much
important to look at Shannon information.

I look at what we need to describe the
evolution of a system. So I'm much more

interested in what kind of model can be
encoded with this type of, with this

information, and how does it correlate to,
or to which degree is it isomorphic or

homomorphic to another system that I want
to model? How much does it model the

observations?
Herald: Thank you. Let's go back to

asking one question, and I would like to
have one question from microphone

number 3.
Q: Thank you for this interesting talk.

My question is really whether you
think that intelligence and this thinking

about a self, or this abstract level of
knowledge are necessarily related.

So can something only be intelligent
if it has abstract thought?

Joscha: No, I think you can make models
without abstract thought, and the majority

of our models are not using abstract
thought, right? Abstract thought is a very

impoverished way of thinking. It's
basically you have this big carpet and you

have a few knitting needles, which are
your abstract thought, and which you can

lift out a few knots in this carpet and
correct them. And the process that form

the carpet are much more rich and
prevalent automatic. So abstract thought

is able to repair perception, but most of
all models are perceptual. And the

capacity to make these models is often
given by instincts and by models outside

the abstract realm. If you have a lot of
abstract thinking it's often an indication

that you use a prosthesis, because some of
your primary modelling is not working very

well. So I suspect that my own models is
largely a result of some defect in my

primary modeling, so some of my instincts
are wrong when I look at the world.

That's why I need to repair my perception
more often than other people. So I have

more abstract ideas on how to do that.
Herald: And we have one question

from our lovely stream observers, stream
watchers, so please a question from the

Internet.
Q: Yeah, I guest this is also related,

partially. Somebody is asking:
How would you suggest to teach your mind

to treat oneself better?

Joscha: So, difficulty is, as soon as you

get access to your source code you can do
bad things. And it's - there are a lot of

techniques to get access to the source
code and then it's dangerous to make them

accessible to you before you know what you
want to have, before you're wise enough to

do this, right? It's like having cookies.
Your - my children think that the reason,

why they don't get all the cookies they
want, is that there is some kind of

resource problem.
<i>laughter</i>

Basically the parents are depriving them
of the cookies that they so richly

deserve. And you can get into the room,
where your brain bakes the cookies. All

the pleasure that you experience, and all
the pain that you experience are signals

that the brain creates for you, right, the
physical world does not create pain.

They're just electrical impulses traveling
through your nerves. The fact that they

mean something is a decision that your
brain makes, and the value, the valence

that gives to them is a decision that you
make. It's not you as a self, it's a

system outside of yourself. So the trick,
if you want to get full control, is that

you get in charge, that you identify with
the mind, with the creator of these

signals. And you don't want to de-
personalize, you don't want to feel that

you become the author of reality, because
that means it's difficult to care about

anything that this organism does. You just
realize "Oh, I'm running on the brain of

that person, but I'm no longer that
person. I can't decide what that person

wants to have, and to do." And that's very
easy to get corrupted or not doing

anything meaningful anymore, right? So,

maybe a good situation for you,
but not a good one for your loved ones.

And meanwhile there are
tricks to get there faster. You can use

rituals, for instance. Shamanic ritual is
something, where, a religious ritual

that powerfully bypasses your self and
talks directly to the mind. And you can

use groups, in which a certain environment
is created, in which a certain behavior

feels natural to you, and your mind
basically gets overwhelmed into adopting

different values and calibrations. So
there are many tricks to make that happen.

What you can also do is you can identify a
particular thing that is wrong and

question yourself "why do I have to suffer
about this?" and you'll become more stoic

about this particular thing and only get
disturbed when you realize actually

it helps to be disturbed about this, and
things change. And with other things you

realize it doesn't have any influence on
how reality works, so why should I have

emotions about this and get agitated? So
sometimes becoming adult means that you

take charge of your own emotions and
identifications.

<i>Applause</i>

Herald: Ok. Let's continue with

microphone number 2 and I think this is
one of the last questions.

Q: So where does pain fit on the
individual and the self-destructive

tendencies on a group level fit in?
Joscha: So in some sense I think that all

consciousness is born over a disagreement
with the way the universe works. Right?

Otherwise you cannot get attention. And
when you go down on this lowest level of

phenomenal experience, in meditation for
instance, and you really focus on this,

what you get is some pain. It's the inside
of a feedback loop that is not at the

target value. Otherwise you don't notice
anything. So pleasure is basically when

this feedback loop gets closer to the
target value. When you don't have a need

you cannot experience pleasure in this
domain. There's this thing that's better

than remarkably good and it's unremarkably
good, it's never been bad. You don't

notice it. Right? So all the pleasure you
experience is because you had a need

before this. You can only enjoy an orgasm
because you have a need for sex that was

unfulfilled before. And so pleasure
doesn't come for free. It's always the

reduction of a pain. And this pain can be
outside of your attention so you don't

notice it and you don't suffer from it.
And it can be a healthy thing to have.

Pain is not intrinsically bad. For the
most part it's a learning signal that

tells you to calibrate things in your
brain differently to perform better. On a

group level, we basically are multi-level
selection species. I don't know if there's

such a thing as group pain. But I also
don't understand groups very well. I see

these weird hive minds but I think it's
basically people emulating what the group

wants. Basically that everybody thinks by
themselves as if they were the group but

it means that they have to constrain what
they think is possible and permissible

to think.

So this feels very unaesthetic to me
and that's why I kind of sort of refuse it.

Haven't found a way to make it
happen in my own mind.

<i>Applause</i>

Joscha: And I suspect many of you
are like this too.

It's like the common condition
in nerds that we have difficulty with

conformance. Not because we want to be
different. We want to belong. But it's

difficult for us to constrain our mind in
the way that it's expected to belong. You

want to be expected, er, be accepted while
being ourself, while being different. Not

for the sake of being different, but
because we are like this. It feels very

strange and corrupt just to adopt because
it would make us belong, right? And this

might be a common trope
among many people here.

<i>Applause</i>

Herald: I think the Q and A and the talk

was equally amazing and I would love to
continue listening to you, Joscha,

explaining the way I work.
Or the way we all work.

<i>audience, Joscha laughing</i>
Herald: That's pretty impressive.

Please give it up, a big round of applause
for Joscha!

<i>Applause</i>

subtitles created by c3subtitles.de
in the year 2019. Join, and help us!