0:00:06.953,0:00:11.186 Hello everyone, and a warm welcome to[br]Multimodal Language Processing. 0:00:11.451,0:00:17.001 My name is Xaver Funk, and I recently had[br]the chance to really [involve] myself into 0:00:17.162,0:00:20.447 this topic, because I am studying[br]neurosciences and this was kind of 0:00:20.447,0:00:26.029 something that I had to do. And, yeah,[br]that's what I want to share with you today. 0:00:26.476,0:00:32.568 So, what I have been doing recently also,[br]is learning arabic, and a little bit of 0:00:32.568,0:00:37.301 mongolian. And mostly what I did was,[br]I had this stream of auditory signals 0:00:37.301,0:00:43.122 that maybe came from the ASML audio, and[br]I tried to match those to symbols that 0:00:43.122,0:00:46.389 were representing these, right, in the[br]book. 0:00:47.472,0:00:52.005 And I kind of had this feeling that this[br]is incomplete. 0:00:52.005,0:00:53.840 So there is something missing there. 0:00:53.840,0:00:59.786 And while I was on the other hand,[br]studying a lot about multimodal language 0:00:59.786,0:01:03.714 processing, which I how gestures influence[br]processing and stuff like that. 0:01:04.571,0:01:07.919 I came to the conclusion that, yeah, there[br]is something missing. 0:01:08.604,0:01:12.269 In our world today, we are all litterate[br]so we mostly think of languages as 0:01:12.269,0:01:16.485 these auditory signals, these mouth noises[br]and the symbols that represent these. 0:01:16.485,0:01:21.585 But there is so much more going on, in[br]face to face communication and, yeah, 0:01:21.585,0:01:25.285 I want to make this point clear, with a[br]virtual experiment. 0:01:25.651,0:01:33.135 So, I want to invite you to first of all,[br]listen to this audio excerpt, from an 0:01:33.135,0:01:38.485 "Easy Languages" video. And I give you the[br]subtitles here, with the english 0:01:38.485,0:01:43.784 translation as well. So, basically, these[br]are auditory signals in Dutch, 0:01:43.784,0:01:47.200 and sequences of symbols[br]in Dutch and English. 0:01:47.200,0:01:51.468 And for the people learning Dutch, please[br]just ignore the English, just to make it 0:01:51.468,0:01:55.200 a little bit harder. And people who know[br]Dutch, please close your eyes, so that 0:01:55.200,0:01:59.700 you don't see it at all.[br]So, let's go. 0:02:40.556,0:02:46.415 So, when I was listening to this at first,[br]I was - because I know some Dutch, 0:02:46.415,0:02:50.342 I was understanding quite a lot,[br]but, kind of, not everything. 0:02:50.342,0:02:54.594 And then, I watched the video that goes[br]with it, it was kind of a different experience. 0:02:54.897,0:03:00.900 And that's what we are going to do now.[br]So just watch the video, and if you can, 0:03:00.900,0:03:07.151 see how these two women, that are[br]interviewed here, are interacting 0:03:07.151,0:03:10.166 with the interviewer, and[br]between each other. 0:03:50.926,0:03:54.007 So, I hoped this worked, and you felt[br]a little bit different now. 0:03:54.007,0:03:57.889 And even for the people who don't know[br]Dutch, I hope you could kind of follow 0:03:57.889,0:04:03.057 what was going on. And even if you[br]didn't, the point I want to make is that 0:04:03.057,0:04:06.290 messages are not only auditory,[br]they are always also visual. 0:04:06.602,0:04:11.186 We have a lot of non-auditory[br]articulators, like 43 face muscles 0:04:11.186,0:04:15.920 for example, and then 2x 34 muscles[br]in the hands, and then even more in 0:04:15.920,0:04:20.891 the arms, in our torso.[br]And the people in this video 0:04:20.891,0:04:24.775 really knew how to use these.[br]So for example we had a lot of 0:04:24.775,0:04:27.991 facial movement going on,[br]like you see on the top, here. 0:04:27.991,0:04:32.191 See how she raise her eyebrows,[br]and then, you have this head tilting 0:04:32.191,0:04:35.189 at the end, that really put [br]an emphasis on what she's saying. 0:04:35.189,0:04:39.190 And then there is a lot of gaze switching[br]as well.That's right in the begining, 0:04:39.190,0:04:42.524 when she says [br](Dutch): Oh genoeg ! Heb je even ? 0:04:42.524,0:04:46.341 So, "Oh, there is so much that[br]I want to see! Do you have some time?" 0:04:46.574,0:04:51.625 But, she doesn't really say "time", she[br]says "Heb je even", "Do you have a little" 0:04:51.625,0:04:55.312 And for me, when I was only listening,[br]I didn't quite get what she was saying, 0:04:55.312,0:04:58.417 but when I saw how she adresses[br]the interviewer, I kind of got it, 0:04:58.417,0:04:59.418 afterwards. 0:05:00.568,0:05:05.918 So then there is of course manual gestures[br]like "hoop op mijn list", 0:05:05.918,0:05:11.669 that's that one here, "hoop op mijn list".[br]She says "berglandschap", so that's 0:05:11.669,0:05:16.884 a mountain range, and then "lang geleden"[br]"long time ago", right? 0:05:16.884,0:05:22.194 So there is a lot of messages that are[br]supported with these manual gestures. 0:05:22.712,0:05:26.593 Then there is also stuff like this[br]nose scratching, where we don't even know 0:05:26.593,0:05:30.294 is there something to it, or is it just[br]a nose scratching. 0:05:30.576,0:05:34.044 Does it carry some information?[br]We don't know. 0:05:35.160,0:05:37.810 And then, lastly also arm and torso[br]movements. 0:05:37.810,0:05:41.144 And also if you watch at the top here,[br]you have nodding. So you see how 0:05:41.144,0:05:45.610 these two kind of nod together, they[br]really give us the impression of 0:05:45.610,0:05:51.094 how good friends they are, right.[br]And then if you look at this bottom part 0:05:51.094,0:05:54.586 here, that's my favorite part of the[br]video. You really have this complex 0:05:54.586,0:06:00.292 orchestration of different gestures,[br]and they are turn-taking. 0:06:00.547,0:06:07.072 So, the one on the right says something,[br]and the one on the left answers that 0:06:07.072,0:06:12.955 perfectly, and then you have gestures,[br]and then the, putting their hair back, 0:06:12.955,0:06:18.471 right, so there is so much going on[br]between them, and it really gives more 0:06:18.471,0:06:21.988 than just the auditory message, right. 0:06:23.153,0:06:27.487 So, note that there is something that[br]our brain has to achieve here. 0:06:27.487,0:06:31.487 Mainly two things : so, it has to[br]segregate all of the stuff that is not 0:06:31.487,0:06:34.331 important for the message.[br]That's the segregation problem. 0:06:34.503,0:06:37.993 From the important stuff, and then,[br]take all the important stuff, all 0:06:37.993,0:06:41.893 the auditory and visuals information and[br]put it together into a coherent message. 0:06:41.950,0:06:45.650 That's our binding problem.[br]And all of this, note, all of this is 0:06:45.650,0:06:49.257 under a really tight time constraint,[br]when you're turn taking, when you're 0:06:49.257,0:06:51.654 having a conversation.[br]And if you say something and 0:06:51.654,0:06:54.608 the other person say something,[br]and there is not that much time 0:06:54.608,0:06:57.418 between turns. And if you need[br]more time, then that also has 0:06:57.418,0:07:01.950 a meaning, right ? If you take time, then[br]that means that you're hesitating 0:07:01.950,0:07:05.152 to answer, maybe there is something[br]going on with you emotionally... 0:07:05.152,0:07:10.212 So you don't want to have that as well.[br]So, yeah, so basically, this is really 0:07:10.212,0:07:12.712 a huge computational problem[br]for your brain. 0:07:13.845,0:07:18.006 And well, how did your brain do?[br]Did you feel the video was more difficult 0:07:18.006,0:07:21.657 than the audio ? Did you understand more,[br]or did you understand less? 0:07:21.813,0:07:25.779 Did you feel more in the scene, maybe?[br]Catching more informations between the lines? 0:07:26.653,0:07:31.496 And well, for me, at least as you might[br]guess, for me it was way easier 0:07:31.496,0:07:34.935 to follow with the video to interpret[br]these gestures. And this is kind of 0:07:34.935,0:07:40.700 a paradox. So, how come that processing[br]more signals simultaneously is easier 0:07:40.700,0:07:45.933 that processing speech alone?[br]And this also was shown 0:07:45.933,0:07:49.232 in the litterature, so people have made[br]experience with this. 0:07:49.732,0:07:54.566 And this is really a surprising[br]facilitation. For example, there are 0:07:54.566,0:07:57.132 lot of studies, I'll just give you[br]one example. 0:07:57.415,0:08:01.116 So in this study they showed people[br]a "prime", so this was some video 0:08:01.116,0:08:05.364 of an action that somebody did, and then[br]they showed the people different videos. 0:08:05.798,0:08:10.348 And the videos were either completely[br]congruent, so what was said was the same 0:08:10.348,0:08:14.349 as the gesture, and was the same as[br]this prime. So in that case it would be 0:08:14.349,0:08:18.614 "chop" and doing the chopping gesture.[br]And then there were different conditions 0:08:18.614,0:08:26.414 where either the speech was congruent,[br]incongruent, and the gesture was congruent. 0:08:26.608,0:08:29.845 Or the speech was incongruent and[br]the gesture was congruent. 0:08:29.947,0:08:34.364 And then they had also weakly congruent[br]stuff, like, this for "chopping", 0:08:34.364,0:08:37.747 but this is actually cutting so this is[br]only weakly incongruent. 0:08:37.747,0:08:41.515 And then this twisting, which is[br]strongly incongruent. 0:08:42.480,0:08:47.263 And then people had to press a button[br]for "yes" if either the speech or 0:08:47.263,0:08:52.133 the gesture were related to the prime,[br]and no if neither speech nor gesture 0:08:52.133,0:08:55.376 was related to the prime.[br]And what the people found out was that 0:08:55.376,0:08:59.050 there were differences in response times,[br]and also in the proportion of errors 0:08:59.050,0:09:02.396 that people did, as soon as soon as [br]there were something incongruent. 0:09:02.396,0:09:07.720 And from that the authors come to[br]the conclusion that really, speech and 0:09:07.720,0:09:11.908 gestures are two sides of the[br]same coin, they mutually interact 0:09:11.908,0:09:16.084 to enhance comprehension.[br]And now the big question is, of course, 0:09:16.084,0:09:19.317 how does our brain achieve this surprising[br]facilitation? 0:09:21.984,0:09:25.800 And we can look back at turn-taking,[br]to maybe get some clues here. 0:09:25.800,0:09:32.049 So on average a turn-take takes only[br]about 0 to 200 miliseconds, which is 0:09:32.049,0:09:38.066 a fifth of a second. You can see in this[br]video how fast she is responding, 0:09:38.066,0:09:43.099 right now, after this, like, this is[br]an instant, right? 0:09:43.861,0:09:47.948 And this is quite extraordinary, because[br]producing a single word actually takes 0:09:47.948,0:09:49.371 about 600 miliseconds. 0:09:49.688,0:09:54.871 So if I just prompt you to say a word,[br]you would take 600 miliseconds to say it. 0:09:55.854,0:10:00.104 So there's something going on, it seems[br]like we are predicting already what we are 0:10:00.104,0:10:04.154 going to say before the turn of the other[br]person is finished, and we already prepare 0:10:04.154,0:10:05.361 our turn. 0:10:05.361,0:10:09.558 So there is something that is going on,[br]that has to do with prediction. 0:10:10.692,0:10:14.993 Most language use in conversation[br]has to be based on prediction somehow. 0:10:14.993,0:10:18.309 And this is quite nice, because prediction[br]is anyways the current hype 0:10:18.309,0:10:21.809 in neuroscience nowadays, and it's[br]basically a good candidate for 0:10:21.809,0:10:23.975 the overarching function of the brain. 0:10:25.296,0:10:30.629 And many people think that what we are[br]doing in our daily lives is basically 0:10:30.629,0:10:34.378 constantly computing and updating[br]probability distributions. 0:10:35.027,0:10:38.534 And this applies both to action,[br]to perception, and also to language. 0:10:39.554,0:10:43.666 So, this will be a rephrasing of[br]the problem we had before, 0:10:44.417,0:10:46.934 as a prediction problem.[br]And this become then, 0:10:46.934,0:10:51.083 "given the preceding context - so, given[br]all the words that come before - what word 0:10:51.083,0:10:56.683 is most likely to come up next?"[br]Right? And to make this more clear, 0:10:56.683,0:11:01.000 let me give you a quick example:[br]so, imagine I come to you and I say, 0:11:01.000,0:11:03.949 without anymore context "I would like to". 0:11:05.033,0:11:07.967 And then, you don't know what I'm going[br]to say next, right? 0:11:07.967,0:11:11.595 It could be any of these, for example.[br]I would like to drink, eat, work... 0:11:11.595,0:11:15.509 And so on.[br]And now, if I shape my hand 0:11:15.509,0:11:20.958 in the form of a "C", and I put it to[br]my mouth, like this, while I say 0:11:20.958,0:11:26.540 "I would like to", then your probability[br]distribution over these words changes 0:11:26.540,0:11:31.133 in such a way that "drink" is much more[br]likely to be the next word. 0:11:31.133,0:11:35.048 And maybe "eat" also a little bit, but[br]the others words probably not, 0:11:35.048,0:11:39.481 because, you can associate this[br]gesture with drinking, or a little bit 0:11:39.481,0:11:43.064 with eating, because it's also[br]something that you put to your mouth, 0:11:43.064,0:11:47.448 but mostly this is commonly understood[br]as "drinking", right. 0:11:48.508,0:11:53.941 So, in this way gestures add context to[br]predictions and help this process of 0:11:53.941,0:11:56.663 predicting, and that also helps the[br]comprehension. 0:11:57.924,0:12:01.489 And, we can actually measure prediction,[br]using neurophysiology. 0:12:02.582,0:12:09.760 So this is EEG, and "EEG" stands for[br]"Electro Encephalography", and 0:12:09.760,0:12:13.343 it's basically putting electrodes on the[br]scalp, and then measuring 0:12:13.343,0:12:20.175 the brain activity that's below.[br]If you do this, well you can measure 0:12:20.175,0:12:23.910 brain activity basically.[br]What people usually do, is that 0:12:23.910,0:12:27.913 they give people these sentences.[br]So these could be normal sentences, 0:12:27.913,0:12:31.460 like this one : "It was his first day[br]at work." 0:12:32.109,0:12:37.065 Or it could be so-called garden-path[br]sentences. So these are sentences that are 0:12:37.065,0:12:42.766 somehow manipulated artificially[br]to elicit some response. Right? 0:12:42.766,0:12:46.499 So this would be : "He spread the warm[br]bread with socks." 0:12:46.499,0:12:51.446 So you may have a weird feeling on[br]your head, because nobody spreads 0:12:51.446,0:12:56.877 the bread with socks. And this weird[br]feeling, if we would measure you with 0:12:56.877,0:13:03.762 an EEG, would constitute this reaction[br]here, that's a so-called N400. 0:13:03.762,0:13:09.946 "N" because it is a negative polarity,[br]and it's 400 miliseconds after the word. 0:13:10.211,0:13:14.494 So, all of this above here is just[br]electrical activity, right? And you have 0:13:14.494,0:13:18.978 this really pronounced peak, when[br]there is a violation of the semantics, 0:13:18.978,0:13:24.669 like with "socks".[br]And it's also taken to be a prediction 0:13:24.669,0:13:28.652 error. So you did not predict socks,[br]you predicted Nutella, for example, 0:13:28.652,0:13:33.187 or honey. But not socks. And this is[br]reflected in this N400 prediction error. 0:13:33.402,0:13:38.219 So people are doing this a lot, like, showing[br]these sentences that are somehow manipulated. 0:13:38.219,0:13:43.119 We have another example here, this is[br]another topic : if you write in all caps 0:13:43.119,0:13:45.926 you have this kind of response for[br]example. 0:13:46.944,0:13:50.560 But, what I want to do now with you[br]is bringing you more to the cutting edge 0:13:50.560,0:13:55.810 of what is currently done in multimodal[br]processing research. 0:13:56.843,0:14:01.042 So the trend is to go away from these[br]artificially constructed sentences, and 0:14:01.042,0:14:05.626 more towards naturalistic language[br]comprehension. So, using actual stories, 0:14:05.626,0:14:11.575 actual sentences, that are not manipulated[br]in any way. And this will be combined with 0:14:11.575,0:14:15.875 computational linguistics - how that[br]works, you will see in a bit. 0:14:16.325,0:14:21.375 And also, yeah, with that you can look at[br]multimodal processing if you just add 0:14:21.375,0:14:26.024 a video to the audio that[br]you make people listen to. 0:14:27.274,0:14:32.691 And what it might look like[br]is like this. So, this is one study 0:14:32.691,0:14:36.274 that is currently not published[br]officially yet. It is already on 0:14:36.274,0:14:43.740 the archive. And I want to use this to[br]illustrate to you how we might research 0:14:43.740,0:14:49.473 naturalistic language comprehension.[br]So the general planners get some 0:14:49.473,0:14:54.407 per-word measures - so these would be[br]these ones here. So for each word, 0:14:54.407,0:14:59.874 there is some value attached.[br]And then we can use these as regressors 0:14:59.874,0:15:04.739 in the big linear regression model.[br]So, using fancy statistics, and with that 0:15:04.739,0:15:12.780 we're basically asking our data "how well[br]are you predicted by these regressors?" 0:15:14.399,0:15:19.281 And for example, this one here is[br]surprise and this is closely related 0:15:19.281,0:15:25.165 to predictions or prediction errors.[br]So this is the negative log probability 0:15:25.165,0:15:28.631 of a word, given all of the words[br]that come before it. 0:15:28.631,0:15:32.962 So, this is the contexte, basically,[br]and this is some word, "w". 0:15:34.111,0:15:37.411 So this is basically telling you how[br]unpredictable is a given word. 0:15:38.560,0:15:42.677 And this measure is base on computational[br]language models, so for example, 0:15:42.677,0:15:47.610 they would take the whole corpus of[br]a language, and then, see which words 0:15:47.610,0:15:54.076 occurs after each other, and thereby get[br]to this value of how unpredictable it is. 0:15:55.960,0:16:00.776 And then, they have another thing here.[br]They use the fundamental frequency of each 0:16:00.776,0:16:05.528 word as a pitch indicator, to control[br]for prosody, which is also pretty cool. 0:16:06.492,0:16:10.659 So they let loose their linear[br]regression models, with these predictors, 0:16:10.659,0:16:17.558 so they have a surprisal value for each[br]word, for example, a prosody, ready for 0:16:17.558,0:16:21.680 each word, then they indicate where[br]there are meaningful gestures happening, 0:16:21.680,0:16:25.408 and, yeah, also, mouth movements. 0:16:26.940,0:16:30.407 And, what came out of this, one finding[br]that might be interesting for us, 0:16:30.407,0:16:36.906 now, is that for meaningful gestures,[br]the N400 is less negative. 0:16:37.806,0:16:42.807 So you can also see this here : for[br]meaningful gesture, this blue line, 0:16:42.807,0:16:47.823 you see that it is a lot less negative[br]than the red line where the gestures are 0:16:47.823,0:16:53.221 absent. And then there's also, that's why[br]I told you about surprisal, an interesting 0:16:53.221,0:16:57.590 interaction between gestures and[br]surprisal. So, the higher the surprisal, 0:16:57.590,0:17:03.072 the less unexpected a word, the stronger[br]this facilitating effect of gestures is. 0:17:04.048,0:17:09.725 Which is also really interesting.[br]Then there's, this is a similar study, 0:17:11.606,0:17:14.873 that I actually got the chance to work on,[br]with a colleague. 0:17:16.550,0:17:21.454 So what we did here, we had a measure of[br]entropy. This measures basically the 0:17:21.521,0:17:25.601 uncertainty about the next word.[br]So if you think back to the example 0:17:25.601,0:17:30.968 we had before, where I was telling[br]you "I would like to", and then something, 0:17:30.968,0:17:34.856 but without context, that would be really[br]high entropy, really high uncertainty : 0:17:34.856,0:17:39.530 you don't know what's coming next. Right?[br]Then we also had surprisal, we had word 0:17:39.530,0:17:43.630 frequency, how often the word came up.[br]And IVC is a measure of, 0:17:43.630,0:17:49.889 it's an abbreviation for "instantaneous[br]visual change", so, how much the actor 0:17:49.889,0:17:56.205 moved while we were showing this to[br]the people. And then speech envelope, 0:17:56.205,0:18:00.773 this is basically a measure of the level[br]of the sound. 0:18:02.662,0:18:08.804 And what we found is - and this[br]by the way was an FRMI experiment, 0:18:08.804,0:18:12.612 so we can look at wich regions are active[br]during some condition. 0:18:13.428,0:18:17.994 And for words where the surprisal was[br]really high, there were these regions 0:18:17.994,0:18:20.991 in red active, and for words where[br]entropy was really high, 0:18:20.991,0:18:25.512 these regions in blue. And now if[br]we look at interactions with gestures 0:18:25.512,0:18:31.812 for the entropy condition, we can see that[br]when there were gestures present, we had 0:18:31.812,0:18:37.790 really specific activations compared[br]to when there were no gestures present, 0:18:37.790,0:18:41.278 in situations where there is high[br]entropy, so high uncertainty. 0:18:42.586,0:18:50.070 So with these tools we try to get into[br]the processes that underlie prediction 0:18:50.070,0:18:55.902 in language.[br]So let's take a step back, and have a look 0:18:55.902,0:19:00.069 at kind of a more global[br]evolutionnary perspective. 0:19:01.670,0:19:06.386 We know from primate research that gesture[br]and gaze are crucial for communication. 0:19:06.769,0:19:10.034 You can see it in this video : this ape[br]right here does this gesture, 0:19:10.464,0:19:14.878 and this signals to its mother[br]to pick her up. Right? 0:19:15.485,0:19:21.970 So these are bonobos, and you can see[br]right now, this "pick me up" gesture. 0:19:23.523,0:19:29.338 And Federico Rossano, from the Max Planck[br]Institute for Evolutionnary Institute, 0:19:30.189,0:19:35.939 could show that this gesture get more[br]and more ritualized, to the point where 0:19:35.939,0:19:45.989 it becomes only a small wrist bend with[br]the arm and one gaze, to instantiate 0:19:45.989,0:19:50.188 this carry behavior. Right?[br]So you see that there is also kind of 0:19:50.188,0:19:56.505 a prediction involved : the mother has[br]to predict what the child is wanting 0:19:56.505,0:20:00.088 to do, right? [br]Going from this, to only this. 0:20:01.170,0:20:06.987 Then, building on this, there are some[br]authors that propose that speech and 0:20:06.987,0:20:11.686 gesture have a common origin.[br]And the idea here is that, through 0:20:11.686,0:20:15.202 these ritualized gestures that[br]we've just seen in those bonobos, 0:20:15.202,0:20:18.653 after a time there will be a proto[br]sign language evolving. 0:20:19.003,0:20:22.136 Which then at some point will be[br]accompanied by sound as well, 0:20:22.136,0:20:27.502 evolving into a proto speech language.[br]And then the proto sign, the proto speech, 0:20:27.502,0:20:32.851 will reinforce each other more and more,[br]until language emerges. 0:20:34.385,0:20:40.470 And another point, here,[br]or an observation : those of you who have 0:20:40.470,0:20:44.138 tried sign language, it kind of feel[br]surprinsingly natural, right? 0:20:44.138,0:20:50.720 So, if speech is the true communication[br]medium for humans, why is it 0:20:50.720,0:20:56.286 thet sign language feels so real,[br]so natural, right? 0:20:58.201,0:21:03.435 And then another point that goes into this[br]theory is that voluntary hand movements 0:21:03.435,0:21:07.551 came before voluntary breathing. And you[br]need voluntary breathing to articulate 0:21:07.551,0:21:08.818 yourself, right? 0:21:09.683,0:21:13.898 So, also, just as complementary to speech, 0:21:13.898,0:21:18.056 you can more easily show spatial[br]relations between things. 0:21:18.969,0:21:22.622 And then, if you look at child development[br]the same pattern : gesture develop 0:21:22.622,0:21:29.885 before speech, and pre-speech turn-taking[br]is faster than later. 0:21:29.885,0:21:34.585 So if you're a baby and you gesture,[br]the turn-taking with your mother, 0:21:34.585,0:21:40.150 the communication is quite fast,[br]it's almost adult level turn-taking. 0:21:40.734,0:21:44.916 Then as you learn language it gets way[br]slower, and only in middle-school it gets 0:21:44.916,0:21:47.950 gets back to the adult level turn-taking. 0:21:48.834,0:21:52.866 So, what's the point, right? What does all[br]of this mean for language learning? 0:21:54.083,0:21:58.783 So for this, let's do another[br]time-travel, back to 1768, 0:21:58.783,0:22:07.332 and meet this French Jesuit monk :[br]Claude-François Lizarde de Radonvilliers. 0:22:07.719,0:22:11.614 And he wrote this book :[br](French) "About The Way To Learn Languages" 0:22:12.407,0:22:17.673 back in the day, where he reflected on how[br]we should teach people languages. 0:22:18.239,0:22:23.672 And interestingly, this is basically[br]the grandfather of the Assimil method, 0:22:24.038,0:22:29.940 and also the Méthode[br]Toussaint-Langenscheidt, or, also called 0:22:29.940,0:22:34.188 "interlinearversion". So this would be[br]this sheet here. 0:22:34.188,0:22:38.521 This was a way people learned languages[br]at the turn of the previous century. 0:22:39.471,0:22:44.856 And you can see here that you have[br]the spanish at the top, then some 0:22:44.856,0:22:50.092 consideration in the middle,[br]and on the bottom the german. 0:22:50.910,0:22:53.959 And this is kind of similar to what[br]Assimil does, right? 0:22:55.176,0:22:58.258 So this is really interesting,[br]but that's not the point here. 0:22:58.988,0:23:03.944 What he also did in this book is to compare[br]L1 - so first language acquisition - 0:23:03.944,0:23:09.240 with second language learning.[br]And he noted that it seems that, 0:23:09.240,0:23:15.739 for the first language, parents show their[br]children pictures, and enact words or 0:23:15.739,0:23:20.089 concepts, and encourage the children to do[br]the same, like this little boy does here. 0:23:20.089,0:23:24.787 But for second language acquisition all we[br]do is give people these vocabulary lists, 0:23:24.787,0:23:27.289 and expect them to learn it[br]just like that. 0:23:28.137,0:23:33.172 So this is kind of an interesting point,[br]and since then it has been shown, 0:23:33.172,0:23:37.422 - and this is actually pretty robust,[br]I was really surprised, that it has been 0:23:37.422,0:23:42.804 a really robust finding, that gesture[br]enriched material enhances learning. 0:23:43.429,0:23:51.882 So in this study, for example, people[br]tried to teach english-speaking people 0:23:51.882,0:23:56.465 japanese words, and they had four[br]different training conditions. 0:23:56.983,0:24:03.065 So one, only speech, one repeated speech,[br]one speech plus incongruent gestures 0:24:03.448,0:24:07.367 - so gestures that would not match -[br]and then, congruent gestures. 0:24:08.095,0:24:10.263 And this is the interesting condition,[br]right? 0:24:11.166,0:24:15.598 And then they tested the people after[br]encoding for three different times : 0:24:15.598,0:24:18.431 after five minutes, after two days,[br]and after one week. 0:24:19.316,0:24:24.900 And also they tested them on forced choice[br]so it's basically multiple choice, 0:24:25.632,0:24:29.215 and free recall, so it's prompting [br]the people with the word, and then they 0:24:29.215,0:24:34.714 come up themselves with the answer.[br]So these numbers here are basically 0:24:34.714,0:24:37.498 the proportion of correct [br]answers that people give. 0:24:37.983,0:24:40.350 And you can see that,[br]across the board, 0:24:40.350,0:24:43.480 the speech plus congruent[br]gesture condition is 0:24:43.480,0:24:50.557 very superior compared to the other ones,[br]which is, yeah, which is interesting, and 0:24:50.557,0:24:57.395 so, you would maybe think that the point[br]is "okay, so we just use videos instead 0:24:57.395,0:24:59.845 of audios", right?[br]And this is what I would call 0:24:59.845,0:25:03.364 Multisensory enrichment.[br]And there is nothing wrong with this, 0:25:03.364,0:25:10.660 this is really useful, you have[br]these YouTube channel like Easy Languages 0:25:10.660,0:25:15.079 - I'm not sponsored by the way (laugh) -[br]where you have conversations with 0:25:15.079,0:25:18.698 real people that from time to time[br]make gestures, and you get the full 0:25:18.698,0:25:23.298 conversation thing, right?[br]And you have these one-on-one videos, 0:25:23.298,0:25:30.415 like this one from Mandarin Corner.[br]Where they are also a lot of gestures 0:25:30.415,0:25:34.096 involved, so the host, Eileen, really[br]tries to integrate a lot of gestures. 0:25:35.147,0:25:38.548 But this is actually not the point[br]- I mean, this is cool but I think 0:25:38.548,0:25:43.614 you already do that.[br]The point is way deeper. 0:25:43.981,0:25:49.813 So, there's another thing going on,[br]not only when you watch gestures, 0:25:49.813,0:25:54.064 but when you enact them.[br]This is called the enactment effect. 0:25:55.113,0:25:58.007 This was actually coined in 1980,[br]by two germans. 0:25:58.318,0:26:03.695 They called it first the "Tu-Effekt",[br]which translates literally to "Do-Effect". 0:26:04.887,0:26:09.103 And you can see why people chose to call[br]it the enactment effect, because it sounds 0:26:09.103,0:26:13.154 way more fancy (laugh) but I really like[br]the "tu-effekt", it sounds funny. 0:26:14.072,0:26:19.521 Anyways, the point is that action words[br]or phrases, this is what they - Engelkamp 0:26:19.521,0:26:23.288 et Krumnacker - noticed : that action[br]words and phrases are remembered better 0:26:23.288,0:26:27.236 if they're acted out,[br]or accompanied by gestures. 0:26:27.539,0:26:33.320 So if you would learn the phrase[br]"chopping garlic", then if you enact it 0:26:33.320,0:26:36.619 actually while learning it,[br]you will retain it way better. 0:26:37.284,0:26:40.247 And this effect is also[br]really well replicated, 0:26:40.247,0:26:42.483 and this was also[br]really surprising to me, 0:26:42.483,0:26:48.896 because it is virtually not at all[br]translated into actual teaching. 0:26:49.190,0:26:53.146 Nobody does this, nobody tells[br]the students to enact things, right? 0:26:53.318,0:26:55.715 Enact words, enact anything. 0:26:55.715,0:27:01.337 And it has been well replicated[br]across tasks, across materials and also 0:27:01.337,0:27:05.419 across populations : across children,[br]adults, even clinical populations : 0:27:05.419,0:27:08.453 People with Alzheimer, people recovering[br]from stroke... 0:27:08.453,0:27:13.053 Somehow people made them[br]learn words and then act the words, 0:27:13.053,0:27:18.235 and it worked better[br]than without enactment. 0:27:19.155,0:27:23.472 And also, this is not only true for[br]action words and concrete words, 0:27:23.472,0:27:26.455 but also abstract words.[br]Anything you can somehow find 0:27:26.455,0:27:31.685 a representation - with gestures - for,[br]you can use this enactment effect. 0:27:32.579,0:27:36.339 And this is way more powerful than[br]multysensory enrichment, 0:27:36.339,0:27:39.940 and we can call this [br]"sensorimotor enrichment", 0:27:39.960,0:27:42.501 because you use[br]your senses and your "motor". 0:27:44.402,0:27:50.567 So, this is also, this ties in with[br]another really interesting development 0:27:50.567,0:27:53.018 in neurosciences, called[br]"embodied cognition". 0:27:53.568,0:27:58.283 Basically this is the idea that many[br]features of cognition - and these might be 0:27:58.283,0:28:01.935 concepts, categories, reasoning or[br]judgement - are shaped by aspects 0:28:01.935,0:28:06.400 of the body. And this would be[br]the motor system - so how we move - 0:28:06.400,0:28:11.266 the perceptual system - what we see, what[br]we feel, what we hear, and so on. 0:28:11.266,0:28:14.517 And also bodily interactions with[br]the environment. 0:28:15.339,0:28:19.336 And you might see where I get with this,[br]if you think about concepts and categories 0:28:19.796,0:28:23.549 What are words, if not concepts and[br]categories, right? 0:28:23.912,0:28:27.579 So we might ask the question, "how are[br]words represented in the brain?" 0:28:28.507,0:28:31.051 And there is this really funny study, 0:28:31.601,0:28:37.760 they showed people words that had strong[br]olfactory associations, which means 0:28:37.810,0:28:42.397 they either stink really hard, or[br]they smell really well. 0:28:43.112,0:28:49.096 And, in case you're looking for some[br]inspiration for your spanish poem, 0:28:49.096,0:28:52.693 you can go (laugh) to this publication and[br]search through the list of words. 0:28:52.693,0:28:57.248 This is also a small - this is only[br]a small sample, there are tons of 0:28:57.248,0:29:02.197 really strong smelling words[br]in the study and, yeah. 0:29:02.530,0:29:06.481 So basically, what they found is that[br]when they showed people these words, 0:29:06.481,0:29:11.797 as compared to words that did not smell[br]that much, some regions in the brain 0:29:11.797,0:29:16.424 that are associated with olfaction,[br]so, with smelling, lighted up. 0:29:17.695,0:29:24.076 And this kind of has been[br]extended as well to actions. 0:29:24.358,0:29:29.339 So, on the left here, these are[br]all the regions that light up 0:29:29.339,0:29:32.473 when you move your foot[br]when you move your fingers, 0:29:32.473,0:29:36.089 or when you move your tongue.[br]And on the right here, these are 0:29:36.089,0:29:42.207 the regions that light up when you read[br]leg-related words, arm-related words, 0:29:42.207,0:29:46.206 or face-related words.[br]And you can see that, this more or less, 0:29:46.206,0:29:50.762 this is more or less,[br]these activations fit each other, right? 0:29:51.105,0:29:55.649 So, in some way, leg-related words[br]are stored where you also move you leg, 0:29:55.649,0:29:59.296 arm-related words are stored where[br]you also move your arms, and so on. 0:30:00.071,0:30:07.139 So, we can think about words actually[br]as functional networks, like this. 0:30:08.388,0:30:12.872 And, note that words are[br]experience-dependent functional networks. 0:30:13.888,0:30:17.139 And experience is connected[br]to the body, right? 0:30:18.254,0:30:25.504 So for exemple, you surely have, not only[br]read and heard the word "garlic", 0:30:25.504,0:30:29.287 you also have smelled garlic, [br]you touched garlic, you tasted garlic 0:30:29.287,0:30:31.544 and, really important thing[br]you chopped garlic. 0:30:31.544,0:30:34.518 So when you read "garlic", [br]you not only have 0:30:34.518,0:30:38.635 the core language areas - in yellow here -[br]activated, but also 0:30:38.635,0:30:44.935 subcortical olfactory areas, and some[br]gustatory areas - so, for taste - 0:30:44.935,0:30:49.302 action areas, right, and visual areas[br]as well. 0:30:49.602,0:30:54.418 So, what I want to tell you here,[br]think about this when you learn languages. 0:30:54.817,0:31:02.428 Did you do the same for "knoblauch", for[br]example - the german word for "garlic"? 0:31:02.688,0:31:09.017 If you learn german, do you actually[br]get into this huge associated network? 0:31:09.978,0:31:13.153 So, and that's the point basically,[br]we are coming to the end, 0:31:13.653,0:31:18.166 the point is that language is multimodal,[br]you should use sensory-motor enrichment 0:31:18.166,0:31:22.347 when learning languages, and thereby[br]embody your languages. 0:31:23.780,0:31:27.413 And if you want to learn more about this,[br]and also for me to give credit, 0:31:27.413,0:31:31.061 this is basically where I got most[br]of my input from. 0:31:31.679,0:31:36.980 These are four big review articles that[br]discuss all of this stuff. 0:31:37.715,0:31:43.729 So yeah, that's basically it.[br]Thanks for listening, and hoping for some cool questions. 0:31:53.427,0:31:57.695 I would most certainly guess so.[br]This thing with the phone is also 0:31:57.695,0:32:01.012 something that I have[br]experienced quite a lot. 0:32:03.928,0:32:09.374 I have lived in Chile for some time,[br]and I've got myself a chilean SIM-card 0:32:09.865,0:32:14.745 And I didn't give this number to a lot[br]of people, but somehow, this number got 0:32:14.745,0:32:18.477 to people that were, I don't know,[br]trying to sell me something. 0:32:19.375,0:32:22.158 And I would get a call from[br]somebody, pick up the phone, 0:32:22.158,0:32:24.198 and I would not understand[br]a single word. 0:32:24.198,0:32:28.108 Like, Chilean spanish is already[br]really hard, and then it's completely 0:32:28.108,0:32:31.337 out of context, I don't know what[br]this person wants from me, and then 0:32:31.337,0:32:34.387 it's just [gestures] and I'm like[br]"sorry, I don't understand you" 0:32:34.387,0:32:36.004 "I don't understand you",[br]"I don't understand you", 0:32:36.004,0:32:37.799 over and over again. 0:32:37.808,0:32:40.995 And yeah, I mean, if you're on the phone, 0:32:42.508,0:32:45.867 there's also a little bit of noise maybe, 0:32:46.177,0:32:50.259 and I really have the feeling that[br]that makes, 0:32:50.259,0:32:52.617 especially in a foreign language, 0:32:52.617,0:32:56.221 conversing that much harder, because[br]you don't see the mouth movements, 0:32:56.821,0:33:01.563 it's not that clear of a sound,[br]you don't see anything else, and yeah. 0:33:01.563,0:33:04.168 I would say so.[br]Cool question. 0:33:17.914,0:33:21.113 I would guess so, I would guess so.[br]Like, I mean, 0:33:22.783,0:33:25.552 especially for autistic people[br]there is a lot of research 0:33:25.798,0:33:28.176 on language processing in general, 0:33:29.577,0:33:34.104 but I don't know of any studies that are 0:33:34.894,0:33:38.419 specifically for multimodal processing, 0:33:38.419,0:33:40.579 but I think there are quite a few. 0:33:41.238,0:33:43.838 Actually the experiment that I showed you, 0:33:47.277,0:33:50.820 the one that I worked on, as well,[br]in the middle of the presentation, 0:33:50.820,0:33:54.991 the entropy stuff, we also did this[br]with schizophrenic patients, 0:33:55.946,0:33:58.020 but we have not looked at the data yet. 0:33:58.020,0:34:00.230 So once this publication is done then, 0:34:01.468,0:34:05.360 somebody else will deal with 0:34:05.360,0:34:07.793 the clinical data,[br]with the schizophrenic people, 0:34:08.151,0:34:10.683 and in general for schizophrenics there's, 0:34:11.282,0:34:12.529 there's a lot of, 0:34:14.528,0:34:19.425 like, language-related abnormalities, 0:34:20.506,0:34:22.826 and I think for autistic people as well. 0:34:23.368,0:34:24.814 I'm not sure about ADHD 0:34:28.364,0:34:31.354 but yeah, it would be[br]a really interesting thing to, 0:34:33.178,0:34:36.289 to look at this for autistic people, for sure. 0:34:36.289,0:34:40.270 And maybe people did this.[br]You can, maybe, you can look it up. 0:34:41.295,0:34:45.939 I don't have anything in my head right now[br]but yeah. 0:34:47.187,0:34:49.696 There should be something. 0:34:59.249,0:35:04.157 I think one of the studies that[br]I glanced over actually tried this. 0:35:04.157,0:35:07.528 So they had some gestures that were nonsense 0:35:08.296,0:35:11.471 I don't know if it was with abstract words[br]or with concrete words, 0:35:11.850,0:35:13.957 but they used nonsense gestures, 0:35:13.957,0:35:16.901 and they still had an effect,[br]but it was smaller. 0:35:17.886,0:35:20.649 So if you try to make this, 0:35:21.098,0:35:22.915 to integrate this into your studies, 0:35:23.548,0:35:28.985 I would suggest that you try to find[br]an enactment that is as sensical as it gets 0:35:29.758,0:35:33.905 I mean, it's not that it's impossible[br]to get enactment for abstract words, 0:35:34.712,0:35:38.378 you just have to be a little bit more[br]creative, and I think the more creative 0:35:38.378,0:35:42.928 you will be the more effective.[br]Like, similar to mnemonics, 0:35:42.928,0:35:48.236 like, the more crazy mnemonic is,[br]the easier it is to remember. 0:35:48.863,0:35:53.014 I could see the same effect with[br]enactments as well. 0:35:53.719,0:35:58.417 And if you should use signs from[br]sign languages, 0:35:59.238,0:36:03.565 I think if you want to that's a cool idea,[br]because then you automatically also learn 0:36:03.565,0:36:07.485 the sign language.[br]And when I was preparing this presentation 0:36:07.485,0:36:11.322 I actually thought about this.[br]Like why do we learn languages? 0:36:13.040,0:36:17.325 Like if I now start learning a language,[br]why do I not learn the sign language 0:36:17.325,0:36:19.313 that goes with it, right? 0:36:19.313,0:36:22.551 I think it would make things easier,[br]because you actually have 0:36:22.551,0:36:24.228 the enactment ready for you, 0:36:25.038,0:36:27.911 and it's just a cool thing, right? 0:36:27.911,0:36:31.963 You can talk with so many more people. 0:36:33.303,0:36:40.171 And also, I think in general, people[br]should learn sign languages regardless 0:36:41.951,0:36:47.419 This became really clear to me actually[br]at the Polyglot Gathering in 2019. 0:36:47.419,0:36:53.206 In Bratislava we were on some ship where[br]there was a party. 0:36:55.074,0:36:58.520 There was loud music, then there were some[br]people who knew sign language 0:36:58.520,0:37:00.469 maybe they are listening right now. 0:37:01.362,0:37:04.407 So they just started, they were like,[br]on the dance floor, 0:37:04.422,0:37:07.814 and instead of screaming into[br]each other's ears, like people usually do, 0:37:07.814,0:37:10.484 they just started to sign, [br]and it was so smooth, like 0:37:11.367,0:37:15.147 why should we communicate with sound[br]when we can do it with gestures. Right? 0:37:15.924,0:37:19.256 I think for many situations[br]it would be a lot easier. 0:37:20.471,0:37:23.809 So yeah, if you can use the signs[br]of your target language, 0:37:24.726,0:37:26.696 I think that's a cool idea. 0:37:31.382,0:37:34.978 Yeah, it does sound like that.[br]Indeed, indeed. 0:37:37.204,0:37:39.763 Yeah, I can totally see that. 0:37:50.576,0:37:55.068 There is a study, that I came across while[br]reasearching, but I didn't look into it. 0:37:55.920,0:38:00.468 If you want the reference, you can[br]reach out to me somehow and I can see 0:38:00.468,0:38:03.013 if I can find it and send it to you. 0:38:05.291,0:38:07.999 I didn't look deeply into it, 0:38:09.514,0:38:13.747 and I think I wouldn't find it now quickly. 0:38:14.977,0:38:19.668 So again, there's something done, but[br]I can't recall it from my head right now. 0:38:23.974,0:38:27.022 Well so, there's two things 0:38:27.704,0:38:30.043 maybe more things but let's start with two 0:38:30.263,0:38:32.844 So first of all when you learn a new word, 0:38:33.664,0:38:37.353 try to get the whole picture of the word. 0:38:38.027,0:38:44.474 Like the garlic example, try to imagine[br]how it smells, how it feels, 0:38:44.474,0:38:46.911 how you chop it, try to enact it. 0:38:47.803,0:38:54.332 Take a moment, and really try to activate[br]the whole functional network of this word. 0:38:55.129,0:39:02.470 And then the other thing was also just[br]to use input with video, 0:39:02.470,0:39:04.472 if you're learning with some input. 0:39:05.121,0:39:08.575 Look up if you find some interesting[br]channels on YouTube or something. 0:39:09.121,0:39:12.838 And then also, that might have not[br]been clear from my presentation, 0:39:12.838,0:39:16.408 if you are conversing with people, use signs. 0:39:20.161,0:39:25.094 I don't know if people do this naturally[br]in general, I think I kind of do it, 0:39:25.094,0:39:29.392 if I talk in my target language,[br]and I'm not sure about a word 0:39:29.392,0:39:32.880 I will try to make sure with my hands[br]that somehow, 0:39:32.880,0:39:36.994 something gets, like, to the other person. 0:39:37.337,0:39:40.130 So what I'm going for is that the other[br]person recognizes 0:39:40.130,0:39:43.005 what I'm trying to say, and then[br]gives me the word, right. 0:39:43.585,0:39:46.006 Like in the example with the glass of water, 0:39:46.514,0:39:48.815 if I don't know "drink" in some language[br]I would, 0:39:48.815,0:39:52.199 I would try to [MIMES][br]Right? "I want to [MIMES]" Right? 0:39:52.242,0:39:57.778 And then have the other person gives me[br]the word, because I'm actively reducing 0:39:57.778,0:40:01.133 the uncertainty that the other perso has,[br]that is trying to predict 0:40:01.133,0:40:04.041 what I'm going to say,[br]by giving gestures. Right? 0:40:05.033,0:40:08.909 So that would be my three[br]practical implications for now. 0:40:18.540,0:40:21.115 Yeah so this is something that[br]I don't know. 0:40:21.905,0:40:27.313 Again I think it's worth trying to do this[br]with the sign language. 0:40:27.313,0:40:31.777 I mean, there's a system of really... 0:40:34.898,0:40:39.082 There's a system of really fitting[br]gestures that people use, 0:40:39.954,0:40:47.100 and it might actually be a good idea[br]to try this out, to use sign language 0:40:47.100,0:40:49.973 as you're learning the actual language, 0:40:51.839,0:40:54.542 to get this enactment working. 0:40:56.030,0:40:59.947 Might be more effective than[br]making up your own gestures. 0:41:01.195,0:41:06.120 I mean if you make up your own gestures[br]you have the advantage that 0:41:09.576,0:41:12.108 during the process of[br]coming up with the gesture, 0:41:12.108,0:41:15.385 you are engaging your brain[br]in a specific way 0:41:15.422,0:41:18.748 that's not there if you just get[br]the gesture from somebody. 0:41:19.615,0:41:21.670 So there might be an advantage there, 0:41:21.670,0:41:27.754 but the other advantage is of course[br]time that you can save, 0:41:27.754,0:41:32.912 and the ability to communicate[br]with people that can't hear 0:41:34.328,0:41:38.154 So yeah, I think that's open for[br]exploration, for sure. 0:41:44.219,0:41:46.336 Well you see it in sign language, right? 0:41:47.940,0:41:51.373 People that sign don't really speak. 0:41:52.394,0:41:55.023 And they get along pretty nicely. 0:41:56.202,0:42:01.003 Another question would be if all of society[br]as a whole can do without verbal. 0:42:01.964,0:42:06.454 That's another question, but I think[br]you can restructure society 0:42:06.454,0:42:10.093 in a way that everybody can communicate[br]with gestures, for sure. 0:42:11.676,0:42:16.894 And according to some people, it was like[br]that before speech developped. 0:42:26.321,0:42:28.821 So yeah, there has been some research,[br]not much. 0:42:29.507,0:42:34.454 If you are interested in this, make sure[br]to check out my presentation on this topic 0:42:34.454,0:42:39.757 from last year's Gathering,[br]and also from last year's conference. 0:42:40.711,0:42:44.512 The conference one is not up on YouTube[br]already, but the Gathering one, 0:42:44.512,0:42:48.932 and there's, in the end, I show some... 0:42:51.711,0:42:55.619 I show a study that was done on[br]polyglots and hyper polyglots 0:42:56.736,0:42:58.669 actually only hyper polyglots I think. 0:43:00.287,0:43:04.837 And so they put people in a FMRI scanner,[br]and just gave them language material. 0:43:06.122,0:43:08.906 And what they found is that[br]the language network 0:43:08.906,0:43:15.690 was less active than for monolinguals. 0:43:16.350,0:43:19.790 So if you listen to something[br]there's some areas 0:43:19.790,0:43:22.082 on the left side of your brain[br]that light up : 0:43:22.082,0:43:26.807 you have some typical areas,[br]like Broca's area, Wernicke's area, 0:43:27.013,0:43:28.584 and some other ones. 0:43:29.003,0:43:34.163 And they found that, for polyglots[br]this "lighting up" is less, 0:43:34.163,0:43:38.823 and the interpretation was that[br]the polyglots' language network, 0:43:38.823,0:43:42.911 through extensive practice, has become[br]more and more efficient 0:43:42.911,0:43:47.581 at dealing with language.[br]And therefore it needs less activation. 0:43:47.976,0:43:53.221 So this is one thing that[br]you observe quite often, 0:43:59.287,0:44:01.908 when there's some process that[br]you get really good at, 0:44:02.673,0:44:05.995 in your brain the activity[br]that you see goes down, 0:44:05.995,0:44:08.331 because the network gets more efficient. 0:44:08.671,0:44:11.354 So that's why this paper was aptly titled 0:44:11.354,0:44:14.955 "The Small And Efficient Network[br]Of Polyglots And Hyperpolyglots". 0:44:15.628,0:44:18.044 And they, you can also look this up as well, 0:44:18.559,0:44:22.382 they also made them listen to[br]different languages, 0:44:23.070,0:44:27.532 and there, the better known[br]the foreign language was 0:44:28.095,0:44:32.391 so the first experiment was completed[br]in english, their mother tongue, 0:44:32.684,0:44:37.113 and then the second experiment they used[br]their target languages like, 0:44:37.113,0:44:39.854 the second best language, third best[br]language and so on. 0:44:40.573,0:44:44.808 And there, the lesser known a language,[br]the less active the language network, 0:44:45.171,0:44:49.040 and the better known, the more active.[br]So you have kind of the opposite effect. 0:44:49.292,0:44:52.473 And they interpreted this as reflecting[br]that the more you know 0:44:52.473,0:44:54.644 in a target language,[br]in a foreign language, 0:44:54.766,0:45:00.342 the more of the language network[br]gets recruited, the more context you have. 0:45:01.707,0:45:05.305 So you have this effect of getting really[br]efficient for your mother tongue, 0:45:05.811,0:45:09.843 and getting more of the whole message for, 0:45:11.579,0:45:13.459 the better you know a foreign language. 0:45:16.014,0:45:18.011 So this was the last question.[br]Alright 0:45:18.766,0:45:23.764 Thanks for listening, thanks to[br]the organizers for organizing this, 0:45:24.106,0:45:27.392 the streaming works really well,[br]I'm really impressed 0:45:27.914,0:45:29.078 Thanks guys!