Frisson Waves - Augmenting Aesthetic Chills in Music Perfromances

Herald: Hello, and welcome back to the FM-
channel. Our next speaker is Kai Konsa.

He's a professor at the Graduate School of
Media Design at Cairo University Japan.

His talk will be about focus on Wirfs.
From what I understood is that it's about

shivers and goosebumps in media
performances that are usually spontaneous.

But somehow he and his team managed to
induce them artificially. But I'm very

forward. I'm looking very forward to this
talk as I don't really understand it. But

now, if you want to ask questions in the
end, RC at the Channel rc3-fm in the

rocket chat at the Channel FM and on
Twitter, and the videos and the hashtag

3FM without the Dash, you can ask
questions that will be answered in the Q&A

session afterwards.
*repated but on english*

Allardyce parking Seotioa your current
idea that some 200 images of a player for

native of translated and that and now I'm
looking forward to a hopefully very

interesting talk.
Konsa: Hello and welcome to my talk

"Frisson Waves", augmenting esthetic
chills in classical music performances.

This is conceptual, early research work
from collaboration of a lot of artists,

designers, researchers, and I'm just
speaker to introduce it to you a little

bit. My name is Kai. And in the next 20
minutes, I will talk to you a little bit

about what is Frisson. Give you a bit of
motivation and background information, why

we are interested in this feeling. And
then I will talk about how can we

recognize into youth and also share
Frisson. And then at the end, I'll talk

about some conclusion and a little bit of
outlook. So the question is, what is

Frisson? You might not have heard the
term, I actually haven't heard Frisson

before we started the research. Two and a
half years, three years ago. But I

definitely knew the feeling. So if you're
listening attentively to a musical piece,

sometimes you might get goose bumps or
some shiver down your spine. And that is

usually triggered from the music. Frisson
is from the French shivers and sorry, my

pronunciation with the German accent. So
you applied for that, I hope. And it's

this psycho physiological phenomenon, that
we feel when we get these goose bumps or

shivers that are triggered from music, but
also other. If you insist and you might

wonder why goosebumps, how can goosebumps
be related to a positive feeling? There is

actually no need to answer theories. I'm
one that I particularly like, is that

Frisson is often induced over music or
over some kind of stimulus that is

repetitive, that has a certain pattern,
and that at one point the pattern breaks

and that surprises you. So this triggers
your autonomous nervous system. So the

fight or flight response you get to
surprise you wonder alertness goes up and

you realize that there's no danger and you
will relax. And feel this esthetic chills.

So the talk, I will give today is an
exploration of the feeling of Frisson with

technology. So how could we detect, induce
or transmit it using especially variable

sensors and actuators and to be a little
bit aclimactic? I can already tell you

that this is still in process. So this is
really exploratory work. However, you

might also wonder why do you care about
this? Why do you want to do this? And you

know, one reason, of course, is. Because
we can and because it's fun, and I think

that's definitely, you know, kind of one
aspect of of the research, however. Also

another reason is so our lab in Yokohama
works in human factors, research, so HCI

and human computer interaction. And we
lately revisited a lot of work also from

cybernetics, also nonlinear dynamics in
terms of research and also in terms of art

and performance. We are very much inspired
by Stilnox work on extending and

augmenting our body. And there's this
realization, if you work on research that,

you know, knowledge is not merely
functional, there's always some kind of

enjoyment in understanding a concept. And
I think also this community will really

understand that type of feeling and this
sense of wonder and this feeling we also

want to explore. We want to understand
ourselves better in terms of cognition

perception, but also in terms of our
feeling. And actually, last year, I gave

also a talk on on boiling mind on an
Frisson loop that we played with and

started researching on. And to some
extent, this expression. If this all beef

talk is just the continuation of this, and
overall, we are also looking for more

creative ways to use physiological data or
other wearable computing sensing that is

not related to surveillance. So extended
to this, we also wonder what does it mean

to be life? It's easy if you think about
transmitting audio video, easy in

quotation marks because yeah, there are
some experts that know a lot about that.

And I see also the effort that goes into
the remote experience and not the

congresses or conferences. However, we
still don't know how to transmit an

atmosphere or a feeling that's much more
difficult. I think the Congress is a very

nice example for that because it moved
from Berlin to Hamburg to Leipzig. But

every time I visited, I kind of felt at
home. I felt, Oh yeah, these are, you

know, kind of the people I like. These are
the culture, the community I belong to,

even though it's at different places. And
we wonder, you know, kind of how can we

transmit at this type of feeling and to
efforts that we get inspired from from

this work is one is neuro life. That's a
project, an EU project with co-

investigator Jamie Ward and also
cybernetic being project here in Japan,

headed by Quarterman Minami. So that deals
with things like parallel agency and

similar. And both of them are actually
also collaborators in the work that I will

present today. So this is the high level
overview why we are interested in this

song, but now getting back to the esthetic
chills. And first, the question is how

could we go about and try to detect or
recognize them? Looking into related work,

of course, we'll see Frisson is that
chills, of course, affect our physiology.

And the first thing that you notice is, of
course, the payload erection. So the goose

bumps that you can get on your arm so the
hairs go up. So we could try to detect

that. However, that might be a little bit
difficult because some people might not

have so much hair on on them and so on. So
then looking into other physiological

changes, respiratory rate is going up for
the sweat glands, electro dermal activity.

You will see more peaks. That's a stress
and excitement indicator, and heart rate

goes up, blood pressure goes up and
usually heart rate variability related

features go down. Because also, if you saw
last year's talk, we already built a

system to record electro dermal activity.
So the sweating on the hand as well as

heart rate we just thought will move along
and use that. Luckily, we also did a

redesign of the wristband bands in the
meantime, so they look a little bit nicer

now and you see also a life demo on my
background right now. So you see EDA and

heart rate behind and if I press here. You
should also see some noise on the sensor.

The visualization, by the way, is done by
Kirill Ragodzin. So thanks for the work!

And then moving forward, so we use these
wristbands to set up a controlled

experiment to detect esthetic chill
events. We just added a trigger, so to add

some self-reporting to it. So in this
case, we really use the user as a self-

report to classify or to label the Frisson
events that has, of course, you know, also

some limitations. So you hope that that's
good enough to capture it. And we used

some music pieces also from related work
and did some counterbalancing and run this

lab study just in know kind of controlled
space or with headphones and so on. We

finished this, but then we also wondered,
you know, how does it look like in real

life in the wild experiments? So we also
organized a concert. With 18 audience

members for one album musical program, and
the set up was the same, so everybody got

a wristband and a trigger. We also added a
third for the pianist, so using EDA from

the foot actually works also relatively
well and then recorded here the data and

hope that people would report their focus
on their esthetic chills. Here's now one

video. A short minute video that shows you
the recording. *piano music* How about the

analysis? I have to see? I'm sorry the
this is still ongoing, so we don't really

have a lot of results yet. And of course,
there were a lot of issues with the life

recording. If you're interested in doing
something similar, contact some of the

technical stuff or also me. We can give
you hints and doing this now over 15 or 20

years and always do something is going
wrong, depending on the setting and so on.

Now I also know more about the classical
music concerts. However, we got some

useful data. The problem there was we
could also train a machine learning model

because we really wanted to dect it real
time. And it seemed to work really well.

We are just still not sure if it really
works or not, so we want to be very

careful about that. So we get higher
accuracy spec. But given the limited

amount of uses we had also and we want to
look into that a little bit more. However,

the analysis, as well as the data sets,
will be publicly available. And if you

want to get them a little bit earlier to
also contact me. So then moving on, this

is the progress on detection. How does it
look like for triggering or inducing

Frisson? So there's also a lot of cool
related work. I just show or highlight two

of them. One is work by Shoko Fukushima at
all. And they're using the electrostatic

effect on the arm to control payload
erection. And they use it to increase the

surprise feeling of somebody, so you put
your arm inside and they can control the

payload erection. Other work is from Ha at
all, whether using three Peltier elements

on your back, on your spine and deactivate
them upwards to also induce Frisson or

static chills. The problem with those two
setups, it's quite hard to get them into a

into a concert hall. And, you know, some
people might not really have much hair on

their arms or so on, so there might be
limitations for it. So then, you know, for

first iteration, we decided to go for a
neck prototype, because kind of the neck

is also a part of some of the Frisson
responses. So you get either chills down

the spine or up the neck or also your hair
might stand up. So we thought it's a good

start and we used first healthy elements
or thermal modules and also vibrant

tactile feedback. In later iterations we
moved just to a thermal feedback to

activate this on the back of the neck
around on the upper side of the trapezius

muscle, and they would activate with
slight cold feedback. So for an initial

tests, it seemed to work or just this,
just with 10 participants, around 30

minutes per participant, we had two music
pieces that are based on related works or

Chopin and Gustav Holst. We counterbalance
the conditions or music pieces with

neckband without neck bent with neck bent,
with activation and without activation.

And from an initial test, we can see that
it seems that slight cold feedback really

provides more instances of reported
Frisson. So there is a slight positive

feedback, but you know, it was still quite
little participants and we'll have to

continue and see also with a little bit of
redesign. So we want to change the order

and placement of the pelty elements for
the continuation work as well. Now moving

to the last part, so we talked about
detection, induction, and now let's talk

about sharing or transmitting Frisson. He
had the idea would be, you know, you are

listening to a musical piece, a classical
piece and one person gets Frisson, does

this detect it over the wristband and then
it's distributed ripples through the

neighbors? They get activated over the
Nick Bend and hopefully also free Frisson

again around the same time it just after
the red circled person felt that the

esthetic chills. So in this case, then,
you know, we would have all of the

audience members need to wear sensors and
actuators, and we would need to freshen

detection and also then the activation
based on that. And for that, we also

organized a not a concert in this case, 50
audience members. The program was around

1,5 hours. And the set up was, as you see
here. So performers on the top, and then

we had two sections. One saw 25 users
would wear just the wristband as a kind of

control group. And the second group, 25
users would wear wristband and neck bent,

so it would get to actually the detection
and also the activation. That's all, you

know, 50 plus wristbands needed charging
and 25 neck bands were manufactured, and

this is the picture from the actual
concert with an nick bent section. And

here is how this should work, so, you
know, you have first one person, you

detect the Frisson and then you ripple it
out to the neighbors, then the next person

might feel Frisson we detected over the
wristband and then replay it out to the

other people that haven't gotten
activation yet and so on. So you have then

a wave of Frisson hopefully moving through
the audience members. This is not a setup

up, yab he (NAME!) who also did a lot of
the organization parts or so on at the

piano, and he is then a small video that
summarizes the work. And at the end you

see also the servers, the recording
server, the activation server and the I

know the detection server and also the
activation server. *piano music playes*

*one person plays piano, another plays
chello* So the question you might have

now, did it work? Hmm. Not not completely
sure. Again, here, work in progress or

analysis is ongoing, and we can't also
really see because, yeah, we had this

control group and we could see more
Frisson events in the sharing group. But

how to interpret that, that's really,
really difficult. We are also working on

the design of the wristbands as well as
the neckband and especially for the

neckband. We got a couple of uses, I think
five or six or seven that really didn't

like the neckband, not the activations or
the slight called activation was OK, but

just because it was a little bit too tight
and a little bit too uncomfortable. So

we're working on a redesign we have for
the next concert in april. All of the data

make it also publicly available. Soon
enough, look also a little bit more what

we can find out about what happened. This
brings me to the end of the presentation.

I hope you enjoyed it. Yeah, I just wanted
to thank a couple of people first and

foremost Yann He, who organized this
Frisson , who introduced us. And also the

teams are dismissed for their second
concert. The extended team thanks a lot

for everybody who was involved here, then
also all of the names. So these are the

people that did the actual work did not
just doing the presenting like I do right

now. I hope I haven't missed anybody. So
also, thanks to George, Dingding, Denny

and so on and all of the other people
involved in group planning the studio

Apollo and also the piano NIST's and
interactive performers. So thanks a lot.

Yeah, that brings me to the end of the
presentation. As I said, we have a third

concert, probably in April next year in
Yokohama, Tokyo area. So if you're

interested, let me know also if you have a
general interest in effect or similar

phenomenon. Also, just write me an email.
It would be good if you mentioned Frisson

remote experience in the subject, so I can
just filter that out and something

completely different. We also have a
conference next year. March submission

deadline is January 7th, augmented humans
in Japan and Germany and cyberspace that

deals with maybe similar work. So thanks a
lot for listening and I hope. Yeah, I told

you something interested in the last 20
minutes. By.

Herald: Hello, and welcome back to the FM
Channel, thank you Kay for the very

interesting talk, and Kay should actually
be with us to answer a few questions.

Hello, Kay.
Kay: Hello.

Herald: And we actually do have a few
questions already. And the first one

sounds a bit more like a comment, but I
tell you anyway. So one one viewer noted

that this a technical cybernetics and
systems course here in Illmenau, the two

in the know and wanted to know if you were
aware of this already.

Kay: Actually, it wasn't, but it sounds
that sounds quite fun and getting more and

more interested in cybernetics as well.
And I think it's useful to revisit some of

the ideas around feedback loops, as I said
at the beginning. So that's cool if you if

you're already looking into that. And I
think especially if people go into HCI

fields, I think it's quite useful to get a
little bit of that background.

Herald: Yes, nice. OK, let's go to the
next question. It's about neural networks,

which have the neural network was us. Have
you considered neural net different doing

differential equations such as echo-state
network or reservoir computing, which are

good when modeling stiff time
consciousness processes?

Kay: That's actually a really good
question and actually also good hint for

what to do next. I just tried to look up.
I saw the question already. Also in the

chat, I try to look up what we used, and I
think at the beginning we just used

support vector machines, so not neural
networks. And no, no, we are using some

neural network, but I don't know the
configuration and I couldn't check. I'll

get back to the person who asked the
question. What was interesting for me was

that the the data looks already quite
good. The sensor data looks actually quite

good. And I would assume that in most
cases, any classifier will do a decent job

for the lab experiments for the other
works. I think, yeah, that sounds quite

interesting. I also want to go more
towards, um yeah, nonlinear dynamics work

as well in terms of of of estimating
Frisson or different feelings. But that's

that's a really good hint, that it would
be more question also for Jo Ann Hunt,

coauthor of the paper that is also linked.
She is our data analyst and so on, and

notes what what she used to the first
classifier was a support vector machine

fairly basic and I think recently we use a
neural network, but I thought it's just

very straightforward. PyTorch long
training, but nothing, nothing special and

nothing fancy so far.
Herald: OK, nice, I guess I guess she will

probably also hear from the questions
then. And then let's go to the next

question. Have you considered or tested
the effects of editorial stimulus, such as

attempting to cross Frisson Waves and in
boring situations instead of like

interesting ones?
Kay: That's also quite a good or

interesting question. I mean, there were
some some audience members also that

mentioned that Nick Bend was actually a
little bit uncomfortable. So I'm not

really sure if we cost Frisson with them.
And I don't know what would happen if you,

I think you probably would just make the
situation uncomfortable anyways. I'm not

sure what would happen then, if you
stimulating called feedback. Actually, you

might get the fear response in these cases
if you're in a boring situation, I'm not

sure if you if you. Hmm. Yeah, I actually
I don't know. It's definitely an

interesting idea to to use it in boring
situations. Can you get somebody to change

their? The feeling and get to the more
excited state? We are playing often, we

played a little bit with the thermal
feedback and it was always interesting if

you change the thermal feedback. So
instead of if you see something hot in VR,

you give cold stimulus or so on. It really
is a little bit confusing and interesting.

I haven't thought about that in Frisson
situation. And if it works for boring

work, but it's definitely cool or
interesting. So if somebody wants to play

with that, I would be up for also giving a
little bit of help or ideas in that

direction.
Herald: OK, thank you for. For answering

these questions, unfortunately, that don't
seem to be any more of them. So that's it.

Thank you very much for the very
interesting talk. That's a topic I haven't

really thinking that much about, but thank
you very much. It was very interesting.

Kay: Thanks a lot also for having me, and
it's always fun and I always enjoyed the

feedback. Yet there's also a candlelight
life demonstration behind me. So you saw

my excitement level kind of increasing or
decreasing with the questions.

Herald: Oh wow, that's pretty cool.
Kay: OK. Yeah, thanks a lot.

Herald: Then here and here on the FM
channel, the next thing happening will at

11:00 p.m. the lightening think talk or
not on the film channel. But one of the

next things happening at three will be at
11:00 p.m. The lightning talks at remote

range. And here on on our channel,
actually at 12:00 a.m. or midnight, there

will be the next terror news show. And
that until then by.