Herald: Hello, and welcome back to the FM-
channel. Our next speaker is Kai Konsa.
He's a professor at the Graduate School of
Media Design at Cairo University Japan.
His talk will be about focus on Wirfs.
From what I understood is that it's about
shivers and goosebumps in media
performances that are usually spontaneous.
But somehow he and his team managed to
induce them artificially. But I'm very
forward. I'm looking very forward to this
talk as I don't really understand it. But
now, if you want to ask questions in the
end, RC at the Channel rc3-fm in the
rocket chat at the Channel FM and on
Twitter, and the videos and the hashtag
3FM without the Dash, you can ask
questions that will be answered in the Q&A
session afterwards.
*repated but on english*
Allardyce parking Seotioa your current
idea that some 200 images of a player for
native of translated and that and now I'm
looking forward to a hopefully very
interesting talk.
Konsa: Hello and welcome to my talk
"Frisson Waves", augmenting esthetic
chills in classical music performances.
This is conceptual, early research work
from collaboration of a lot of artists,
designers, researchers, and I'm just
speaker to introduce it to you a little
bit. My name is Kai. And in the next 20
minutes, I will talk to you a little bit
about what is Frisson. Give you a bit of
motivation and background information, why
we are interested in this feeling. And
then I will talk about how can we
recognize into youth and also share
Frisson. And then at the end, I'll talk
about some conclusion and a little bit of
outlook. So the question is, what is
Frisson? You might not have heard the
term, I actually haven't heard Frisson
before we started the research. Two and a
half years, three years ago. But I
definitely knew the feeling. So if you're
listening attentively to a musical piece,
sometimes you might get goose bumps or
some shiver down your spine. And that is
usually triggered from the music. Frisson
is from the French shivers and sorry, my
pronunciation with the German accent. So
you applied for that, I hope. And it's
this psycho physiological phenomenon, that
we feel when we get these goose bumps or
shivers that are triggered from music, but
also other. If you insist and you might
wonder why goosebumps, how can goosebumps
be related to a positive feeling? There is
actually no need to answer theories. I'm
one that I particularly like, is that
Frisson is often induced over music or
over some kind of stimulus that is
repetitive, that has a certain pattern,
and that at one point the pattern breaks
and that surprises you. So this triggers
your autonomous nervous system. So the
fight or flight response you get to
surprise you wonder alertness goes up and
you realize that there's no danger and you
will relax. And feel this esthetic chills.
So the talk, I will give today is an
exploration of the feeling of Frisson with
technology. So how could we detect, induce
or transmit it using especially variable
sensors and actuators and to be a little
bit aclimactic? I can already tell you
that this is still in process. So this is
really exploratory work. However, you
might also wonder why do you care about
this? Why do you want to do this? And you
know, one reason, of course, is. Because
we can and because it's fun, and I think
that's definitely, you know, kind of one
aspect of of the research, however. Also
another reason is so our lab in Yokohama
works in human factors, research, so HCI
and human computer interaction. And we
lately revisited a lot of work also from
cybernetics, also nonlinear dynamics in
terms of research and also in terms of art
and performance. We are very much inspired
by Stilnox work on extending and
augmenting our body. And there's this
realization, if you work on research that,
you know, knowledge is not merely
functional, there's always some kind of
enjoyment in understanding a concept. And
I think also this community will really
understand that type of feeling and this
sense of wonder and this feeling we also
want to explore. We want to understand
ourselves better in terms of cognition
perception, but also in terms of our
feeling. And actually, last year, I gave
also a talk on on boiling mind on an
Frisson loop that we played with and
started researching on. And to some
extent, this expression. If this all beef
talk is just the continuation of this, and
overall, we are also looking for more
creative ways to use physiological data or
other wearable computing sensing that is
not related to surveillance. So extended
to this, we also wonder what does it mean
to be life? It's easy if you think about
transmitting audio video, easy in
quotation marks because yeah, there are
some experts that know a lot about that.
And I see also the effort that goes into
the remote experience and not the
congresses or conferences. However, we
still don't know how to transmit an
atmosphere or a feeling that's much more
difficult. I think the Congress is a very
nice example for that because it moved
from Berlin to Hamburg to Leipzig. But
every time I visited, I kind of felt at
home. I felt, Oh yeah, these are, you
know, kind of the people I like. These are
the culture, the community I belong to,
even though it's at different places. And
we wonder, you know, kind of how can we
transmit at this type of feeling and to
efforts that we get inspired from from
this work is one is neuro life. That's a
project, an EU project with co-
investigator Jamie Ward and also
cybernetic being project here in Japan,
headed by Quarterman Minami. So that deals
with things like parallel agency and
similar. And both of them are actually
also collaborators in the work that I will
present today. So this is the high level
overview why we are interested in this
song, but now getting back to the esthetic
chills. And first, the question is how
could we go about and try to detect or
recognize them? Looking into related work,
of course, we'll see Frisson is that
chills, of course, affect our physiology.
And the first thing that you notice is, of
course, the payload erection. So the goose
bumps that you can get on your arm so the
hairs go up. So we could try to detect
that. However, that might be a little bit
difficult because some people might not
have so much hair on on them and so on. So
then looking into other physiological
changes, respiratory rate is going up for
the sweat glands, electro dermal activity.
You will see more peaks. That's a stress
and excitement indicator, and heart rate
goes up, blood pressure goes up and
usually heart rate variability related
features go down. Because also, if you saw
last year's talk, we already built a
system to record electro dermal activity.
So the sweating on the hand as well as
heart rate we just thought will move along
and use that. Luckily, we also did a
redesign of the wristband bands in the
meantime, so they look a little bit nicer
now and you see also a life demo on my
background right now. So you see EDA and
heart rate behind and if I press here. You
should also see some noise on the sensor.
The visualization, by the way, is done by
Kirill Ragodzin. So thanks for the work!
And then moving forward, so we use these
wristbands to set up a controlled
experiment to detect esthetic chill
events. We just added a trigger, so to add
some self-reporting to it. So in this
case, we really use the user as a self-
report to classify or to label the Frisson
events that has, of course, you know, also
some limitations. So you hope that that's
good enough to capture it. And we used
some music pieces also from related work
and did some counterbalancing and run this
lab study just in know kind of controlled
space or with headphones and so on. We
finished this, but then we also wondered,
you know, how does it look like in real
life in the wild experiments? So we also
organized a concert. With 18 audience
members for one album musical program, and
the set up was the same, so everybody got
a wristband and a trigger. We also added a
third for the pianist, so using EDA from
the foot actually works also relatively
well and then recorded here the data and
hope that people would report their focus
on their esthetic chills. Here's now one
video. A short minute video that shows you
the recording. *piano music* How about the
analysis? I have to see? I'm sorry the
this is still ongoing, so we don't really
have a lot of results yet. And of course,
there were a lot of issues with the life
recording. If you're interested in doing
something similar, contact some of the
technical stuff or also me. We can give
you hints and doing this now over 15 or 20
years and always do something is going
wrong, depending on the setting and so on.
Now I also know more about the classical
music concerts. However, we got some
useful data. The problem there was we
could also train a machine learning model
because we really wanted to dect it real
time. And it seemed to work really well.
We are just still not sure if it really
works or not, so we want to be very
careful about that. So we get higher
accuracy spec. But given the limited
amount of uses we had also and we want to
look into that a little bit more. However,
the analysis, as well as the data sets,
will be publicly available. And if you
want to get them a little bit earlier to
also contact me. So then moving on, this
is the progress on detection. How does it
look like for triggering or inducing
Frisson? So there's also a lot of cool
related work. I just show or highlight two
of them. One is work by Shoko Fukushima at
all. And they're using the electrostatic
effect on the arm to control payload
erection. And they use it to increase the
surprise feeling of somebody, so you put
your arm inside and they can control the
payload erection. Other work is from Ha at
all, whether using three Peltier elements
on your back, on your spine and deactivate
them upwards to also induce Frisson or
static chills. The problem with those two
setups, it's quite hard to get them into a
into a concert hall. And, you know, some
people might not really have much hair on
their arms or so on, so there might be
limitations for it. So then, you know, for
first iteration, we decided to go for a
neck prototype, because kind of the neck
is also a part of some of the Frisson
responses. So you get either chills down
the spine or up the neck or also your hair
might stand up. So we thought it's a good
start and we used first healthy elements
or thermal modules and also vibrant
tactile feedback. In later iterations we
moved just to a thermal feedback to
activate this on the back of the neck
around on the upper side of the trapezius
muscle, and they would activate with
slight cold feedback. So for an initial
tests, it seemed to work or just this,
just with 10 participants, around 30
minutes per participant, we had two music
pieces that are based on related works or
Chopin and Gustav Holst. We counterbalance
the conditions or music pieces with
neckband without neck bent with neck bent,
with activation and without activation.
And from an initial test, we can see that
it seems that slight cold feedback really
provides more instances of reported
Frisson. So there is a slight positive
feedback, but you know, it was still quite
little participants and we'll have to
continue and see also with a little bit of
redesign. So we want to change the order
and placement of the pelty elements for
the continuation work as well. Now moving
to the last part, so we talked about
detection, induction, and now let's talk
about sharing or transmitting Frisson. He
had the idea would be, you know, you are
listening to a musical piece, a classical
piece and one person gets Frisson, does
this detect it over the wristband and then
it's distributed ripples through the
neighbors? They get activated over the
Nick Bend and hopefully also free Frisson
again around the same time it just after
the red circled person felt that the
esthetic chills. So in this case, then,
you know, we would have all of the
audience members need to wear sensors and
actuators, and we would need to freshen
detection and also then the activation
based on that. And for that, we also
organized a not a concert in this case, 50
audience members. The program was around
1,5 hours. And the set up was, as you see
here. So performers on the top, and then
we had two sections. One saw 25 users
would wear just the wristband as a kind of
control group. And the second group, 25
users would wear wristband and neck bent,
so it would get to actually the detection
and also the activation. That's all, you
know, 50 plus wristbands needed charging
and 25 neck bands were manufactured, and
this is the picture from the actual
concert with an nick bent section. And
here is how this should work, so, you
know, you have first one person, you
detect the Frisson and then you ripple it
out to the neighbors, then the next person
might feel Frisson we detected over the
wristband and then replay it out to the
other people that haven't gotten
activation yet and so on. So you have then
a wave of Frisson hopefully moving through
the audience members. This is not a setup
up, yab he (NAME!) who also did a lot of
the organization parts or so on at the
piano, and he is then a small video that
summarizes the work. And at the end you
see also the servers, the recording
server, the activation server and the I
know the detection server and also the
activation server. *piano music playes*
*one person plays piano, another plays
chello* So the question you might have
now, did it work? Hmm. Not not completely
sure. Again, here, work in progress or
analysis is ongoing, and we can't also
really see because, yeah, we had this
control group and we could see more
Frisson events in the sharing group. But
how to interpret that, that's really,
really difficult. We are also working on
the design of the wristbands as well as
the neckband and especially for the
neckband. We got a couple of uses, I think
five or six or seven that really didn't
like the neckband, not the activations or
the slight called activation was OK, but
just because it was a little bit too tight
and a little bit too uncomfortable. So
we're working on a redesign we have for
the next concert in april. All of the data
make it also publicly available. Soon
enough, look also a little bit more what
we can find out about what happened. This
brings me to the end of the presentation.
I hope you enjoyed it. Yeah, I just wanted
to thank a couple of people first and
foremost Yann He, who organized this
Frisson , who introduced us. And also the
teams are dismissed for their second
concert. The extended team thanks a lot
for everybody who was involved here, then
also all of the names. So these are the
people that did the actual work did not
just doing the presenting like I do right
now. I hope I haven't missed anybody. So
also, thanks to George, Dingding, Denny
and so on and all of the other people
involved in group planning the studio
Apollo and also the piano NIST's and
interactive performers. So thanks a lot.
Yeah, that brings me to the end of the
presentation. As I said, we have a third
concert, probably in April next year in
Yokohama, Tokyo area. So if you're
interested, let me know also if you have a
general interest in effect or similar
phenomenon. Also, just write me an email.
It would be good if you mentioned Frisson
remote experience in the subject, so I can
just filter that out and something
completely different. We also have a
conference next year. March submission
deadline is January 7th, augmented humans
in Japan and Germany and cyberspace that
deals with maybe similar work. So thanks a
lot for listening and I hope. Yeah, I told
you something interested in the last 20
minutes. By.
Herald: Hello, and welcome back to the FM
Channel, thank you Kay for the very
interesting talk, and Kay should actually
be with us to answer a few questions.
Hello, Kay.
Kay: Hello.
Herald: And we actually do have a few
questions already. And the first one
sounds a bit more like a comment, but I
tell you anyway. So one one viewer noted
that this a technical cybernetics and
systems course here in Illmenau, the two
in the know and wanted to know if you were
aware of this already.
Kay: Actually, it wasn't, but it sounds
that sounds quite fun and getting more and
more interested in cybernetics as well.
And I think it's useful to revisit some of
the ideas around feedback loops, as I said
at the beginning. So that's cool if you if
you're already looking into that. And I
think especially if people go into HCI
fields, I think it's quite useful to get a
little bit of that background.
Herald: Yes, nice. OK, let's go to the
next question. It's about neural networks,
which have the neural network was us. Have
you considered neural net different doing
differential equations such as echo-state
network or reservoir computing, which are
good when modeling stiff time
consciousness processes?
Kay: That's actually a really good
question and actually also good hint for
what to do next. I just tried to look up.
I saw the question already. Also in the
chat, I try to look up what we used, and I
think at the beginning we just used
support vector machines, so not neural
networks. And no, no, we are using some
neural network, but I don't know the
configuration and I couldn't check. I'll
get back to the person who asked the
question. What was interesting for me was
that the the data looks already quite
good. The sensor data looks actually quite
good. And I would assume that in most
cases, any classifier will do a decent job
for the lab experiments for the other
works. I think, yeah, that sounds quite
interesting. I also want to go more
towards, um yeah, nonlinear dynamics work
as well in terms of of of estimating
Frisson or different feelings. But that's
that's a really good hint, that it would
be more question also for Jo Ann Hunt,
coauthor of the paper that is also linked.
She is our data analyst and so on, and
notes what what she used to the first
classifier was a support vector machine
fairly basic and I think recently we use a
neural network, but I thought it's just
very straightforward. PyTorch long
training, but nothing, nothing special and
nothing fancy so far.
Herald: OK, nice, I guess I guess she will
probably also hear from the questions
then. And then let's go to the next
question. Have you considered or tested
the effects of editorial stimulus, such as
attempting to cross Frisson Waves and in
boring situations instead of like
interesting ones?
Kay: That's also quite a good or
interesting question. I mean, there were
some some audience members also that
mentioned that Nick Bend was actually a
little bit uncomfortable. So I'm not
really sure if we cost Frisson with them.
And I don't know what would happen if you,
I think you probably would just make the
situation uncomfortable anyways. I'm not
sure what would happen then, if you
stimulating called feedback. Actually, you
might get the fear response in these cases
if you're in a boring situation, I'm not
sure if you if you. Hmm. Yeah, I actually
I don't know. It's definitely an
interesting idea to to use it in boring
situations. Can you get somebody to change
their? The feeling and get to the more
excited state? We are playing often, we
played a little bit with the thermal
feedback and it was always interesting if
you change the thermal feedback. So
instead of if you see something hot in VR,
you give cold stimulus or so on. It really
is a little bit confusing and interesting.
I haven't thought about that in Frisson
situation. And if it works for boring
work, but it's definitely cool or
interesting. So if somebody wants to play
with that, I would be up for also giving a
little bit of help or ideas in that
direction.
Herald: OK, thank you for. For answering
these questions, unfortunately, that don't
seem to be any more of them. So that's it.
Thank you very much for the very
interesting talk. That's a topic I haven't
really thinking that much about, but thank
you very much. It was very interesting.
Kay: Thanks a lot also for having me, and
it's always fun and I always enjoyed the
feedback. Yet there's also a candlelight
life demonstration behind me. So you saw
my excitement level kind of increasing or
decreasing with the questions.
Herald: Oh wow, that's pretty cool.
Kay: OK. Yeah, thanks a lot.
Herald: Then here and here on the FM
channel, the next thing happening will at
11:00 p.m. the lightening think talk or
not on the film channel. But one of the
next things happening at three will be at
11:00 p.m. The lightning talks at remote
range. And here on on our channel,
actually at 12:00 a.m. or midnight, there
will be the next terror news show. And
that until then by.