"What is multimodality?"

Berit Hendriksen and Gunther Kress discuss the notions of 'functional specialisation', 'functional load', 'coherence' and 'layout'.

So what happens when we put modes
together?

Well, behind it all is of course the notion of communication.

I want to communicate something to you
I want to have,

I have a meaning which I want you
to get in some way,

And I have the sense that one mode alone won't do it. And so

put modes together because I have a
sense that this mode will allow me to do

this kind of thing best, and this mode
will allow me to do this other thing

best. So for instance, on this website
that we have here, I can click on a

particular thing and what I have is
something which is in time. There's a

mode of moving image or little video
which is moving image and it's speech.

Together these things are in time,
but at the same time, I have things which

are not in time, bits of writing. So I have
things which are not in time and things

which are and temporarily instantiated.

I have an image, a still image, which is
spatial. And so because I want to say

things which are about the things
which are in time, they move, things which

are not in time, the stationary, things
which are spatially kind of displayed

like images, I want to put all these
together because of the meaning that I

want to make. So I have a sense
of what these things do,

I have a sense of what I want to mean,
and so, I have a kind of an interest to make

a composition in which certain kinds of things come together in a particular way that best

exemplifies or communicates what I want to
mean.

Okay.

So, these modes are kind of arranged according to the interest

I have as the person communicates, but also
my sense of who you are

and what we might be most
interesting, most readily memorable...

but most pleasurable, most informative
for you. So these, these things are sort

of set in the communicational frame.

So, you seem to suggest that different modes can do different kinds of things?

Yes, and I suggested in part that it's me who makes the decision about which mode to use.

There is also of course around me a
sort of conventional notion.

I said for instance, if I'm kind of
communicating something very formally

then writing might seem to be, because
my society tells me, the best means for doing that.

It seems that we have, in our society, not
just these modes available to use, but

also conventions that indicate to us not
as strict rules, nobody's going to get shot,

but as rules, conventions to use
them in particular ways. And so we might

say that writing over many hundreds of
years in literate societies has become specialized

to be the carrier of certain kinds of
information. There's a functional specialization of writing

and, also, also of course, then what happens
is a writing in that past which is now

changing very rapidly, carried most of the informational load, so we can...we can ask, in a particular

on this website, which mode is carrying
most informational load?

So, if we look at this BBC News website,

Yeah.

We can talk about which of the different modes has the

most functional specialization, or functional load?

We can say which mode has been specialized for what kinds of purposes.

Okay.

Yeah, so, the writing next to the image seems
to be specialized to kind of make certain

kinds of things in the image seem more salient.

Mm-hmm

Or, to give a context, a frame, for what this image actually

is. That seems to be what writing is doing here. That's his specialization.

Now, the question is, which of them carries more informational load?

And, I would find that actually quite difficult to
answer because writing kind of frames,

it provides an environment, it provides a
kind of a.. a background to the image.

But the image seems most potent. And then
they're also organized in a kind of a

left-right relationship. Yeah, the image
is on the left and the writing is on the right.

So I'm asking myself, does that
matter? And, would it matter if the

writing was on the left and the image on
the right? Now, these are the kinds of

questions we now have to begin to think
about because we're making a composition

in which everything that is there ought
to be where it should be in terms of

further meanings I want to make.

So, is this what we mean by layout then?

In relation to the module we're just
looking at, we're looking at a module

here of image and writing. And on the
website, it's quite near the top and it's

on the... if this was sort of seen as two
columns really, it's in the left column

that has a particular place there,
seemingly. If you kind of look at the

website, you might come to that
first, especially to scroll down.

You don't have to scroll down in order to
see that. So, something about the space

has been used to make that most
prominent. It's what you come to first

after you've got the BBC band at the top,
its first, it's on the left.

In the West we're trained to read from
the left to the right, so we look there first,

it's most prominent. Layout, the
possibility of putting things in space

in a meaningful way, has been used to
make us attend to that before we attend

to other kinds of things. We don't have
to scroll down. It's kind of prominently there.

That's what we mean by layout.

Is this related to... to if we talk about ordering?

And, absolutely. For instance on this website,
as you can see, there's a...

a smaller module, opposite in
the right column, and where this

particular module about these soldiers
and the tank in the war in Afghanistan are.

And a smaller and kind of thing and...
and somehow... and if you now imagined this

smaller element being on the left and
the module about the war in Afghanistan

being on the right, intuitively, you might
think it would look unbalanced.

It would, it would seem kind of odd. It
wouldn't seem quite right, so there is a

kind of thing about ordering which might
be both about significance, it might be

about a sort of a sense of a proper
composition in which

things should be kind of balanced in terms of
a visual kind of firm ordering of them.

And yes, that's what's all orderings about but
also the fact that as you scroll down,

other things appear. Somebody has thought things which are less in importance

can be lower in the in terms of scrolling down
on the website.

So, the things which you mention earlier in the word "salient," so what they are assumed

being more salient will be for example on the
left, topside?

I would say so, yeah. So in the module,
the tank is on the left the

writing is on the right. In these two columns,

the item on the war in Afghanistan is on the left and the other item which I can't quite

see what it's about, it's trailing something
in some other bit of program,

is on on the right hand side. So there is a kind of a sense

of which are the more significant spaces
on that screen.

In Linguistics, were you two talking about
coherence and cohesion in a text.

How can we use those terms when we're talking about Multimodality?

The way that in my own understanding of Linguistics

that's been used is that cohesion tells
you about the devices you use

for making something coherent. And I think you can use that much in the same way here.

You get a sense, for instance, if this screen was arranged in such a way

that you say sometimes what's most, what seems to be most important

is on the right, sometimes what's most important is on the left

that would produce a sense of
lack of coherence. And the cohesive

device of ordering in space would have
been not really fully used. But also if

for instance, you had a color scheme
here, whether you kind of I don't know

what you call the BBC red there,
a maroon sort of firm red.

Banner.

In that banner. And if that
color.. it could be matched by

other kinds of red's or pink's or green's or blue's and so then the sense of coherence

of the page would be destroyed by kind
of an inconsiderate use of the mode of color.

Yeah, so, there are devices here that produce the sense of yes, this belongs here.

It is where it ought to be.
And this is where it ought to be and

they are of various kinds. Its
positioning, it's.. there is something

coherent about the color that occurs
here, the... the color in the image of the

tank isn't out of place in relation to
the maroon of the banner of the.. the

yeah, so, I think one can transfer the notion
of firm cohesive devices, what are the

means of making something coherent? And,
the notion of it is coherent it makes

sense from the linguistic or from other
things to this. But we would have to sort

of say it's no longer about
sentences in sequence or elements of

sentences, which kind of link across from
sentence to sentence.

But it's different.. different kinds of things. For instance in Linguistics,

nobody would think about color as being a means for making something coherent.