Herald: Hello, and welcome back to the FM- channel. Our next speaker is Kai Konsa.
He's a professor at the Graduate School of Media Design at Cairo University Japan.
His talk will be about focus on Wirfs. From what I understood is that it's about
shivers and goosebumps in media performances that are usually spontaneous.
But somehow he and his team managed to induce them artificially. But I'm very
forward. I'm looking very forward to this talk as I don't really understand it. But
now, if you want to ask questions in the end, RC at the Channel rc3-fm in the
rocket chat at the Channel FM and on Twitter, and the videos and the hashtag
3FM without the Dash, you can ask questions that will be answered in the Q&A
session afterwards. *repated but on english*
Allardyce parking Seotioa your current idea that some 200 images of a player for
native of translated and that and now I'm looking forward to a hopefully very
interesting talk. Konsa: Hello and welcome to my talk
"Frisson Waves", augmenting esthetic chills in classical music performances.
This is conceptual, early research work from collaboration of a lot of artists,
designers, researchers, and I'm just speaker to introduce it to you a little
bit. My name is Kai. And in the next 20 minutes, I will talk to you a little bit
about what is Frisson. Give you a bit of motivation and background information, why
we are interested in this feeling. And then I will talk about how can we
recognize into youth and also share Frisson. And then at the end, I'll talk
about some conclusion and a little bit of outlook. So the question is, what is
Frisson? You might not have heard the term, I actually haven't heard Frisson
before we started the research. Two and a half years, three years ago. But I
definitely knew the feeling. So if you're listening attentively to a musical piece,
sometimes you might get goose bumps or some shiver down your spine. And that is
usually triggered from the music. Frisson is from the French shivers and sorry, my
pronunciation with the German accent. So you applied for that, I hope. And it's
this psycho physiological phenomenon, that we feel when we get these goose bumps or
shivers that are triggered from music, but also other. If you insist and you might
wonder why goosebumps, how can goosebumps be related to a positive feeling? There is
actually no need to answer theories. I'm one that I particularly like, is that
Frisson is often induced over music or over some kind of stimulus that is
repetitive, that has a certain pattern, and that at one point the pattern breaks
and that surprises you. So this triggers your autonomous nervous system. So the
fight or flight response you get to surprise you wonder alertness goes up and
you realize that there's no danger and you will relax. And feel this esthetic chills.
So the talk, I will give today is an exploration of the feeling of Frisson with
technology. So how could we detect, induce or transmit it using especially variable
sensors and actuators and to be a little bit aclimactic? I can already tell you
that this is still in process. So this is really exploratory work. However, you
might also wonder why do you care about this? Why do you want to do this? And you
know, one reason, of course, is. Because we can and because it's fun, and I think
that's definitely, you know, kind of one aspect of of the research, however. Also
another reason is so our lab in Yokohama works in human factors, research, so HCI
and human computer interaction. And we lately revisited a lot of work also from
cybernetics, also nonlinear dynamics in terms of research and also in terms of art
and performance. We are very much inspired by Stilnox work on extending and
augmenting our body. And there's this realization, if you work on research that,
you know, knowledge is not merely functional, there's always some kind of
enjoyment in understanding a concept. And I think also this community will really
understand that type of feeling and this sense of wonder and this feeling we also
want to explore. We want to understand ourselves better in terms of cognition
perception, but also in terms of our feeling. And actually, last year, I gave
also a talk on on boiling mind on an Frisson loop that we played with and
started researching on. And to some extent, this expression. If this all beef
talk is just the continuation of this, and overall, we are also looking for more
creative ways to use physiological data or other wearable computing sensing that is
not related to surveillance. So extended to this, we also wonder what does it mean
to be life? It's easy if you think about transmitting audio video, easy in
quotation marks because yeah, there are some experts that know a lot about that.
And I see also the effort that goes into the remote experience and not the
congresses or conferences. However, we still don't know how to transmit an
atmosphere or a feeling that's much more difficult. I think the Congress is a very
nice example for that because it moved from Berlin to Hamburg to Leipzig. But
every time I visited, I kind of felt at home. I felt, Oh yeah, these are, you
know, kind of the people I like. These are the culture, the community I belong to,
even though it's at different places. And we wonder, you know, kind of how can we
transmit at this type of feeling and to efforts that we get inspired from from
this work is one is neuro life. That's a project, an EU project with co-
investigator Jamie Ward and also cybernetic being project here in Japan,
headed by Quarterman Minami. So that deals with things like parallel agency and
similar. And both of them are actually also collaborators in the work that I will
present today. So this is the high level overview why we are interested in this
song, but now getting back to the esthetic chills. And first, the question is how
could we go about and try to detect or recognize them? Looking into related work,
of course, we'll see Frisson is that chills, of course, affect our physiology.
And the first thing that you notice is, of course, the payload erection. So the goose
bumps that you can get on your arm so the hairs go up. So we could try to detect
that. However, that might be a little bit difficult because some people might not
have so much hair on on them and so on. So then looking into other physiological
changes, respiratory rate is going up for the sweat glands, electro dermal activity.
You will see more peaks. That's a stress and excitement indicator, and heart rate
goes up, blood pressure goes up and usually heart rate variability related
features go down. Because also, if you saw last year's talk, we already built a
system to record electro dermal activity. So the sweating on the hand as well as
heart rate we just thought will move along and use that. Luckily, we also did a
redesign of the wristband bands in the meantime, so they look a little bit nicer
now and you see also a life demo on my background right now. So you see EDA and
heart rate behind and if I press here. You should also see some noise on the sensor.
The visualization, by the way, is done by Kirill Ragodzin. So thanks for the work!
And then moving forward, so we use these wristbands to set up a controlled
experiment to detect esthetic chill events. We just added a trigger, so to add
some self-reporting to it. So in this case, we really use the user as a self-
report to classify or to label the Frisson events that has, of course, you know, also
some limitations. So you hope that that's good enough to capture it. And we used
some music pieces also from related work and did some counterbalancing and run this
lab study just in know kind of controlled space or with headphones and so on. We
finished this, but then we also wondered, you know, how does it look like in real
life in the wild experiments? So we also organized a concert. With 18 audience
members for one album musical program, and the set up was the same, so everybody got
a wristband and a trigger. We also added a third for the pianist, so using EDA from
the foot actually works also relatively well and then recorded here the data and
hope that people would report their focus on their esthetic chills. Here's now one
video. A short minute video that shows you the recording. *piano music* How about the
analysis? I have to see? I'm sorry the this is still ongoing, so we don't really
have a lot of results yet. And of course, there were a lot of issues with the life
recording. If you're interested in doing something similar, contact some of the
technical stuff or also me. We can give you hints and doing this now over 15 or 20
years and always do something is going wrong, depending on the setting and so on.
Now I also know more about the classical music concerts. However, we got some
useful data. The problem there was we could also train a machine learning model
because we really wanted to dect it real time. And it seemed to work really well.
We are just still not sure if it really works or not, so we want to be very
careful about that. So we get higher accuracy spec. But given the limited
amount of uses we had also and we want to look into that a little bit more. However,
the analysis, as well as the data sets, will be publicly available. And if you
want to get them a little bit earlier to also contact me. So then moving on, this
is the progress on detection. How does it look like for triggering or inducing
Frisson? So there's also a lot of cool related work. I just show or highlight two
of them. One is work by Shoko Fukushima at all. And they're using the electrostatic
effect on the arm to control payload erection. And they use it to increase the
surprise feeling of somebody, so you put your arm inside and they can control the
payload erection. Other work is from Ha at all, whether using three Peltier elements
on your back, on your spine and deactivate them upwards to also induce Frisson or
static chills. The problem with those two setups, it's quite hard to get them into a
into a concert hall. And, you know, some people might not really have much hair on
their arms or so on, so there might be limitations for it. So then, you know, for
first iteration, we decided to go for a neck prototype, because kind of the neck
is also a part of some of the Frisson responses. So you get either chills down
the spine or up the neck or also your hair might stand up. So we thought it's a good
start and we used first healthy elements or thermal modules and also vibrant
tactile feedback. In later iterations we moved just to a thermal feedback to
activate this on the back of the neck around on the upper side of the trapezius
muscle, and they would activate with slight cold feedback. So for an initial
tests, it seemed to work or just this, just with 10 participants, around 30
minutes per participant, we had two music pieces that are based on related works or
Chopin and Gustav Holst. We counterbalance the conditions or music pieces with
neckband without neck bent with neck bent, with activation and without activation.
And from an initial test, we can see that it seems that slight cold feedback really
provides more instances of reported Frisson. So there is a slight positive
feedback, but you know, it was still quite little participants and we'll have to
continue and see also with a little bit of redesign. So we want to change the order
and placement of the pelty elements for the continuation work as well. Now moving
to the last part, so we talked about detection, induction, and now let's talk
about sharing or transmitting Frisson. He had the idea would be, you know, you are
listening to a musical piece, a classical piece and one person gets Frisson, does
this detect it over the wristband and then it's distributed ripples through the
neighbors? They get activated over the Nick Bend and hopefully also free Frisson
again around the same time it just after the red circled person felt that the
esthetic chills. So in this case, then, you know, we would have all of the
audience members need to wear sensors and actuators, and we would need to freshen
detection and also then the activation based on that. And for that, we also
organized a not a concert in this case, 50 audience members. The program was around
1,5 hours. And the set up was, as you see here. So performers on the top, and then
we had two sections. One saw 25 users would wear just the wristband as a kind of
control group. And the second group, 25 users would wear wristband and neck bent,
so it would get to actually the detection and also the activation. That's all, you
know, 50 plus wristbands needed charging and 25 neck bands were manufactured, and
this is the picture from the actual concert with an nick bent section. And
here is how this should work, so, you know, you have first one person, you
detect the Frisson and then you ripple it out to the neighbors, then the next person
might feel Frisson we detected over the wristband and then replay it out to the
other people that haven't gotten activation yet and so on. So you have then
a wave of Frisson hopefully moving through the audience members. This is not a setup
up, yab he (NAME!) who also did a lot of the organization parts or so on at the
piano, and he is then a small video that summarizes the work. And at the end you
see also the servers, the recording server, the activation server and the I
know the detection server and also the activation server. *piano music playes*
*one person plays piano, another plays chello* So the question you might have
now, did it work? Hmm. Not not completely sure. Again, here, work in progress or
analysis is ongoing, and we can't also really see because, yeah, we had this
control group and we could see more Frisson events in the sharing group. But
how to interpret that, that's really, really difficult. We are also working on
the design of the wristbands as well as the neckband and especially for the
neckband. We got a couple of uses, I think five or six or seven that really didn't
like the neckband, not the activations or the slight called activation was OK, but
just because it was a little bit too tight and a little bit too uncomfortable. So
we're working on a redesign we have for the next concert in april. All of the data
make it also publicly available. Soon enough, look also a little bit more what
we can find out about what happened. This brings me to the end of the presentation.
I hope you enjoyed it. Yeah, I just wanted to thank a couple of people first and
foremost Yann He, who organized this Frisson , who introduced us. And also the
teams are dismissed for their second concert. The extended team thanks a lot
for everybody who was involved here, then also all of the names. So these are the
people that did the actual work did not just doing the presenting like I do right
now. I hope I haven't missed anybody. So also, thanks to George, Dingding, Denny
and so on and all of the other people involved in group planning the studio
Apollo and also the piano NIST's and interactive performers. So thanks a lot.
Yeah, that brings me to the end of the presentation. As I said, we have a third
concert, probably in April next year in Yokohama, Tokyo area. So if you're
interested, let me know also if you have a general interest in effect or similar
phenomenon. Also, just write me an email. It would be good if you mentioned Frisson
remote experience in the subject, so I can just filter that out and something
completely different. We also have a conference next year. March submission
deadline is January 7th, augmented humans in Japan and Germany and cyberspace that
deals with maybe similar work. So thanks a lot for listening and I hope. Yeah, I told
you something interested in the last 20 minutes. By.
Herald: Hello, and welcome back to the FM Channel, thank you Kay for the very
interesting talk, and Kay should actually be with us to answer a few questions.
Hello, Kay. Kay: Hello.
Herald: And we actually do have a few questions already. And the first one
sounds a bit more like a comment, but I tell you anyway. So one one viewer noted
that this a technical cybernetics and systems course here in Illmenau, the two
in the know and wanted to know if you were aware of this already.
Kay: Actually, it wasn't, but it sounds that sounds quite fun and getting more and
more interested in cybernetics as well. And I think it's useful to revisit some of
the ideas around feedback loops, as I said at the beginning. So that's cool if you if
you're already looking into that. And I think especially if people go into HCI
fields, I think it's quite useful to get a little bit of that background.
Herald: Yes, nice. OK, let's go to the next question. It's about neural networks,
which have the neural network was us. Have you considered neural net different doing
differential equations such as echo-state network or reservoir computing, which are
good when modeling stiff time consciousness processes?
Kay: That's actually a really good question and actually also good hint for
what to do next. I just tried to look up. I saw the question already. Also in the
chat, I try to look up what we used, and I think at the beginning we just used
support vector machines, so not neural networks. And no, no, we are using some
neural network, but I don't know the configuration and I couldn't check. I'll
get back to the person who asked the question. What was interesting for me was
that the the data looks already quite good. The sensor data looks actually quite
good. And I would assume that in most cases, any classifier will do a decent job
for the lab experiments for the other works. I think, yeah, that sounds quite
interesting. I also want to go more towards, um yeah, nonlinear dynamics work
as well in terms of of of estimating Frisson or different feelings. But that's
that's a really good hint, that it would be more question also for Jo Ann Hunt,
coauthor of the paper that is also linked. She is our data analyst and so on, and
notes what what she used to the first classifier was a support vector machine
fairly basic and I think recently we use a neural network, but I thought it's just
very straightforward. PyTorch long training, but nothing, nothing special and
nothing fancy so far. Herald: OK, nice, I guess I guess she will
probably also hear from the questions then. And then let's go to the next
question. Have you considered or tested the effects of editorial stimulus, such as
attempting to cross Frisson Waves and in boring situations instead of like
interesting ones? Kay: That's also quite a good or
interesting question. I mean, there were some some audience members also that
mentioned that Nick Bend was actually a little bit uncomfortable. So I'm not
really sure if we cost Frisson with them. And I don't know what would happen if you,
I think you probably would just make the situation uncomfortable anyways. I'm not
sure what would happen then, if you stimulating called feedback. Actually, you
might get the fear response in these cases if you're in a boring situation, I'm not
sure if you if you. Hmm. Yeah, I actually I don't know. It's definitely an
interesting idea to to use it in boring situations. Can you get somebody to change
their? The feeling and get to the more excited state? We are playing often, we
played a little bit with the thermal feedback and it was always interesting if
you change the thermal feedback. So instead of if you see something hot in VR,
you give cold stimulus or so on. It really is a little bit confusing and interesting.
I haven't thought about that in Frisson situation. And if it works for boring
work, but it's definitely cool or interesting. So if somebody wants to play
with that, I would be up for also giving a little bit of help or ideas in that
direction. Herald: OK, thank you for. For answering
these questions, unfortunately, that don't seem to be any more of them. So that's it.
Thank you very much for the very interesting talk. That's a topic I haven't
really thinking that much about, but thank you very much. It was very interesting.
Kay: Thanks a lot also for having me, and it's always fun and I always enjoyed the
feedback. Yet there's also a candlelight life demonstration behind me. So you saw
my excitement level kind of increasing or decreasing with the questions.
Herald: Oh wow, that's pretty cool. Kay: OK. Yeah, thanks a lot.
Herald: Then here and here on the FM channel, the next thing happening will at
11:00 p.m. the lightening think talk or not on the film channel. But one of the
next things happening at three will be at 11:00 p.m. The lightning talks at remote
range. And here on on our channel, actually at 12:00 a.m. or midnight, there
will be the next terror news show. And that until then by.