1
00:00:00,000 --> 00:00:14,595
34C3 preroll music
2
00:00:14,595 --> 00:00:17,870
Herald: Please give a warm welcome here.
3
00:00:17,870 --> 00:00:23,610
It’s Franziska, Teresa, and Judith.
4
00:00:23,610 --> 00:00:26,510
Judith, you have the stage, thank you.
5
00:00:26,510 --> 00:00:28,470
Judith Hartstein: Thank you, thanks!
6
00:00:28,470 --> 00:00:33,710
applause
7
00:00:33,710 --> 00:00:57,430
inaudible
8
00:00:57,430 --> 00:01:00,219
Judith: We believe that
scientific performance indicators
9
00:01:00,219 --> 00:01:03,269
are widely applied to inform
funding decisions and to
10
00:01:03,269 --> 00:01:08,270
determine the availability of career
opportunities. So, those of you who are
11
00:01:08,270 --> 00:01:14,420
working in science or have had a look into
the science system might agree to that.
12
00:01:14,420 --> 00:01:18,840
And we want to understand evaluative
bibliometrics as algorithmic science
13
00:01:18,840 --> 00:01:27,640
evaluation instruments to highlight some
things that do occur also with other
14
00:01:27,640 --> 00:01:38,020
algorithmic instruments of evaluation. And
so we’re going to start with a quote from
15
00:01:38,020 --> 00:01:44,679
a publication in 2015 which reads “As the
tyranny of bibliometrics tightens its
16
00:01:44,679 --> 00:01:49,130
grip, it is having a disastrous effect on
the model of science presented to young
17
00:01:49,130 --> 00:01:58,999
researchers.” We have heard the talk of
hanno already, and he’s basically also
18
00:01:58,999 --> 00:02:06,679
talking about problems in the science
system and the reputation by the
19
00:02:06,679 --> 00:02:14,170
indicators. And the question is, is
bibliometrics the bad guy here? If you
20
00:02:14,170 --> 00:02:18,530
speak of ‘tyranny of bibliometrics’, who
is the actor doing this? Or are maybe
21
00:02:18,530 --> 00:02:24,950
bibliometricians the problem? We want to
contextualize our talk into the growing
22
00:02:24,950 --> 00:02:30,440
movement of Reflexive Metrics. So those
who are doing science studies, social
23
00:02:30,440 --> 00:02:34,940
studies of science, scientometrics and
bibliometrics. The movement of Reflexive
24
00:02:34,940 --> 00:02:41,690
Metrics. So the basic idea is to say:
“Okay, we have to accept accountability if
25
00:02:41,690 --> 00:02:45,540
we do bibliometrics and scientometrics.”
We have to understand the effects of
26
00:02:45,540 --> 00:02:54,220
algorithmic evaluation on science, and we
will try not to be the bad guy. And the
27
00:02:54,220 --> 00:03:03,640
main mediator of the science evaluation
which is perceived by the researchers is
28
00:03:03,640 --> 00:03:09,630
the algorithm. I will hand over the
microphone to… or I will not hand over the
29
00:03:09,630 --> 00:03:14,440
microphone but I will hand over the talk
to Teresa. She’s going to talk about
30
00:03:14,440 --> 00:03:19,690
"Datafication of Scientific Evaluation".
31
00:03:19,690 --> 00:03:24,180
Teresa Isigkeit: Okay. I hope you can
hear me. No? Yes? Okay.
32
00:03:24,180 --> 00:03:25,890
Judith: mumbling
33
00:03:25,890 --> 00:03:29,480
When we think about the science system
what do we expect?
34
00:03:29,480 --> 00:03:33,700
What can society expect
from a scientific system?
35
00:03:33,700 --> 00:03:38,200
In general, we would say
reliable and truthful knowledge,
36
00:03:38,200 --> 00:03:41,810
that is scrutinized by
the scientific community.
37
00:03:41,810 --> 00:03:44,320
So where can we find this knowledge?
38
00:03:44,320 --> 00:03:47,140
Normally in publications.
39
00:03:47,140 --> 00:03:51,920
So with these publications,
can we actually say
40
00:03:51,920 --> 00:03:58,670
whether science is bad or good? Or is
there better science than others?
41
00:03:58,670 --> 00:04:03,350
In the era of
digital publication databases,
42
00:04:03,350 --> 00:04:07,280
there’s big datasets of publications.
43
00:04:07,280 --> 00:04:12,330
And these are used to
evaluate and calculate
44
00:04:12,330 --> 00:04:16,840
the quality of scientific output.
45
00:04:16,840 --> 00:04:22,720
So in general, with this metadata
we can tell you
46
00:04:22,720 --> 00:04:26,400
who is the author of a publication,
47
00:04:26,400 --> 00:04:30,020
where is the home institution
of this author,
48
00:04:30,020 --> 00:04:37,960
or which types of citations are in
the bibliographic information.
49
00:04:37,960 --> 00:04:44,890
This is used in the calculation
of bibliometric indicators.
50
00:04:44,890 --> 00:04:51,580
For example if you take the
journal impact factors,
51
00:04:51,580 --> 00:04:57,650
which is a citation based indicator,
you can compare different journals.
52
00:04:57,650 --> 00:05:03,750
And maybe say which journals
are performing better than others
53
00:05:03,750 --> 00:05:09,360
or if the journal factor has increased or
decreased over the years.
54
00:05:09,360 --> 00:05:15,330
Another example would be the
Hirsch-Index for individual scientists,
55
00:05:15,330 --> 00:05:22,670
which is also widely used when
scientists apply for jobs. So they put
56
00:05:22,670 --> 00:05:27,710
these numbers in their CVs and supposedly
this tells you something about the quality
57
00:05:27,710 --> 00:05:36,270
of research those scientists are
conducting. With the availability of the
58
00:05:36,270 --> 00:05:45,590
data we can see an increase in its usage.
And in a scientific environment in which
59
00:05:45,590 --> 00:05:52,260
data-driven science is established,
scientific conduct decisions regarding
60
00:05:52,260 --> 00:06:04,419
hiring or funding heavily rely on these
indicators. There’s maybe a naive belief
61
00:06:04,419 --> 00:06:12,020
that these indicators that are data-driven
and rely on data that is collected in the
62
00:06:12,020 --> 00:06:26,610
database is a more objective metric that
we can use. So here's a quote by Rieder
63
00:06:26,610 --> 00:06:32,430
and Simon: “In this brave new world trust
no longer resides in the integrity of
64
00:06:32,430 --> 00:06:38,710
individual truth-tellers or the veracity
of prestigious institutions, but is placed
65
00:06:38,710 --> 00:06:44,480
in highly formalized procedures enacted
through disciplined self-restraint.
66
00:06:44,480 --> 00:06:53,150
Numbers cease to be supplements.” So we
see a change of an evaluation system that
67
00:06:53,150 --> 00:07:00,449
is relying on expert knowledge to a system
of algorithmic science evaluation. In this
68
00:07:00,449 --> 00:07:05,699
change there’s a belief in a
depersonalization of the system and the
69
00:07:05,699 --> 00:07:15,090
perception of algorithms as the rule of
law. So when looking at the interaction
70
00:07:15,090 --> 00:07:26,010
between the algorithm and scientists we
can tell that this relationship is not as
71
00:07:26,010 --> 00:07:35,169
easy as it seems. Algorithms are not in
fact objective. They carry social meaning
72
00:07:35,169 --> 00:07:43,020
and human agency. They are used to
construct a reality and algorithms don’t
73
00:07:43,020 --> 00:07:48,290
come naturally. They don’t grow on trees
and can be picked by scientists and people
74
00:07:48,290 --> 00:07:55,120
who evaluate the scientific system, so we
have to be reflective and think about
75
00:07:55,120 --> 00:08:04,910
which social meanings the algorithm holds.
So when there is a code that the algorithm
76
00:08:04,910 --> 00:08:11,169
uses, there is a subjective meaning in
this code, and there is agency in this
77
00:08:11,169 --> 00:08:17,370
code, and you can’t just say, oh, this is
a perfect construction of the reality of
78
00:08:17,370 --> 00:08:22,350
scientific system. So the belief that this
tells you more about the quality of
79
00:08:22,350 --> 00:08:31,820
research is not a good indicator. So when
you think about the example of citation
80
00:08:31,820 --> 00:08:37,049
counts the algorithm reads the
bibliographic information of a publication
81
00:08:37,049 --> 00:08:47,230
from the database. So scientists, they
cite papers that relate to their studies.
82
00:08:47,230 --> 00:08:55,740
But we don’t actually know which of these
citations are more meaningful than others,
83
00:08:55,740 --> 00:09:01,319
so they’re not as easily comparable. But
the algorithms give you the belief they
84
00:09:01,319 --> 00:09:11,230
are, so relevance is not as easily put
into an algorithm and there is different
85
00:09:11,230 --> 00:09:19,199
types of citations. So the scientists
perceive this use of the algorithms also
86
00:09:19,199 --> 00:09:24,790
as a powerful instrument. And so the
algorithm has some sway above the
87
00:09:24,790 --> 00:09:30,019
scientists because they rely so much on
those indicators to further their careers,
88
00:09:30,019 --> 00:09:38,069
to get a promotion, or get funding for
their next research projects. So we have a
89
00:09:38,069 --> 00:09:42,970
reciprocal relationship between the
algorithm and the scientists, and this
90
00:09:42,970 --> 00:09:52,399
creates a new construction of reality. So
we can conclude that governance by
91
00:09:52,399 --> 00:09:59,019
algorithms leads to behavioral adaptation
in scientists, and one of these examples
92
00:09:59,019 --> 00:10:08,069
that uses the Science Citation Index will
be given from Franziska.
93
00:10:08,069 --> 00:10:12,610
Franziska Sörgel: Thanks for the
handover! Yes, let me start.
94
00:10:12,610 --> 00:10:16,070
I’m focusing on reputation
and authorship as you can see
95
00:10:16,070 --> 00:10:21,420
on the slide, and first let me
start with a quote
96
00:10:21,420 --> 00:10:27,490
by Eugene Garfield, which says: “Is it
reasonable to assume that if I cite a
97
00:10:27,490 --> 00:10:32,889
paper that I would probably be interested
in those papers which subsequently cite it
98
00:10:32,889 --> 00:10:38,630
as well as my own paper. Indeed, I have
observed on several occasions that people
99
00:10:38,630 --> 00:10:45,399
preferred to cite the articles I had cited
rather than cite me! It would seem to me
100
00:10:45,399 --> 00:10:51,321
that this is the basis for the building up
of the ‘logical network’ for the Citation
101
00:10:51,321 --> 00:11:01,709
Index service.” So, actually, this Science
Citation Index which is described here was
102
00:11:01,709 --> 00:11:07,970
mainly developed in order to solve the
problems of information retrieval. Eugene
103
00:11:07,970 --> 00:11:16,259
Garfield, also founder of this Science
Citation Index – short: SCI – noted or
104
00:11:16,259 --> 00:11:22,170
began to note a huge interest in
reciprocal publication behavior. He
105
00:11:22,170 --> 00:11:27,470
recognized the increasing interest as a
strategic instrument to exploit
106
00:11:27,470 --> 00:11:33,410
intellectual property. And indeed, the
interest in the SCI – and its data –
107
00:11:33,410 --> 00:11:39,189
successively became more relevant within
the disciplines, and its usage extended.
108
00:11:39,189 --> 00:11:45,800
Later, [Derek J.] de Solla Price, another
social scientist, asked or claimed for a
109
00:11:45,800 --> 00:11:53,029
better research on the topic, as it
currently also meant a crisis in science,
110
00:11:53,029 --> 00:11:59,329
and stated: “If a paper was cited once,
it would get cited again and
111
00:11:59,329 --> 00:12:05,089
again, so the main problem was that the
rich would get richer”, which is also
112
00:12:05,089 --> 00:12:12,029
known as the “Matthew Effect”. Finally,
the SCI and its use turned into a system
113
00:12:12,029 --> 00:12:17,860
which was and still is used as a
reciprocal citation system, and became a
114
00:12:17,860 --> 00:12:25,119
central and global actor. Once a paper was
cited, the probability it was cited again
115
00:12:25,119 --> 00:12:30,720
was higher, and it would even extend its
own influence on a certain topic within
116
00:12:30,720 --> 00:12:38,369
the scientific field. So it was known that
you would either read a certain article
117
00:12:38,369 --> 00:12:48,999
and people would do research on a certain
topic or subject. So this phenomenon would
118
00:12:48,999 --> 00:12:58,569
rise to an instrument of disciplining
science and created power structures.
119
00:12:58,569 --> 00:13:04,530
Let me show you one example which is
closely connected to this phenomenon
120
00:13:04,530 --> 00:13:10,920
I just told you about – and I don’t know
if here in this room there are any
121
00:13:10,920 --> 00:13:18,680
astronomers or physicists?
Yeah, there are few, okay.
122
00:13:18,680 --> 00:13:24,580
That’s great, actually.
So in the next slide, here,
123
00:13:24,580 --> 00:13:32,990
we have a table with a time
window from 2010 to 2016, and social
124
00:13:32,990 --> 00:13:41,069
scientists from Berlin found out that the
co-authorship within the field of physics
125
00:13:41,069 --> 00:13:50,519
extended by 58 on a yearly basis in this
time window. So this is actually already
126
00:13:50,519 --> 00:13:55,920
very high, but they also found another
very extreme case. They found one paper
127
00:13:55,920 --> 00:14:07,399
which had roundabout 7,000 words and the
mentioned authorship of 5,000. So, in
128
00:14:07,399 --> 00:14:15,389
average, the contribution of each
scientist or researcher of this paper who
129
00:14:15,389 --> 00:14:28,910
was mentioned was 1.1 word. Sounds
strange, yeah. And so of course you have
130
00:14:28,910 --> 00:14:34,519
to see this in a certain context, and
maybe we can talk about this later on,
131
00:14:34,519 --> 00:14:41,380
because it has to do with Atlas particle
detector, which requires high maintenance
132
00:14:41,380 --> 00:14:46,019
and stuff. But still, the number of
authorship, and you can see this
133
00:14:46,019 --> 00:14:52,519
regardless which scientific field we are
talking about, generally increased the
134
00:14:52,519 --> 00:15:05,360
last years. It remains a problem
especially for the reputation, obviously.
135
00:15:05,360 --> 00:15:12,399
It remains a problem that there is such
high pressure on nowadays researchers.
136
00:15:12,399 --> 00:15:20,200
Still, of course, we have ethics and
research requires standards of
137
00:15:20,200 --> 00:15:25,689
responsibility. And for example there’s
one, there’s other ones, but there’s one
138
00:15:25,689 --> 00:15:30,850
here on the slide: the “Australian Code
for the Responsible Conduct of Research”
139
00:15:30,850 --> 00:15:36,689
which says: “The right to authorship is
not tied to position or profession and
140
00:15:36,689 --> 00:15:41,060
does not depend on whether the
contribution was paid for or voluntary.
141
00:15:41,060 --> 00:15:46,409
It is not enough to have provided
materials or routine technical support,
142
00:15:46,419 --> 00:15:50,969
or to have made the measurements
on which the publication is based.
143
00:15:50,969 --> 00:15:55,129
Substantial intellectual involvement
is required.”
144
00:15:55,129 --> 00:16:03,259
So yeah, this is, could be one rule
to work with or to work by, to follow.
145
00:16:03,259 --> 00:16:08,079
And still we have this problem
of reputation which remains,
146
00:16:08,079 --> 00:16:11,480
and where I hand over to Judith again.
147
00:16:11,480 --> 00:16:19,739
Judith: Thank you. So we’re going to speak
about strategic citation now. So if you
148
00:16:19,739 --> 00:16:29,540
put this point of reputation like that,
you may say: So the researcher does find
149
00:16:29,540 --> 00:16:36,239
something in his research, his or her
research, and addresses the publication
150
00:16:36,239 --> 00:16:40,049
describing it to the community. And the
community, the scientific community
151
00:16:40,049 --> 00:16:46,269
rewards the researcher with reputation.
And now the algorithm, which is like
152
00:16:46,269 --> 00:16:54,730
perceived to be a new thing, is mediating
the visibility of the researcher’s results
153
00:16:54,730 --> 00:17:00,720
to the community, and is also mediating
the rewards – the career opportunities or
154
00:17:00,720 --> 00:17:04,919
the funding decisions etc. And what
happens now and what is plausible to
155
00:17:04,919 --> 00:17:10,398
happen is that the researcher addresses
his or her research also to the algorithm
156
00:17:10,398 --> 00:17:21,039
in terms of citing those who are evaluated
by the algorithm, who he wants to support,
157
00:17:21,039 --> 00:17:29,230
and also in terms of strategic keywording
etc. And that’s the only thing which
158
00:17:29,230 --> 00:17:33,809
happens new, might be a perspective on
that. So the one thing new: the algorithm
159
00:17:33,809 --> 00:17:40,700
is addressed as a recipient of scientific
publications. And it is like far-fetched
160
00:17:40,700 --> 00:17:46,260
to discriminate between so-called and
‘visible colleges’ and ‘citation cartels’.
161
00:17:46,260 --> 00:17:50,899
What do I mean by that? So ‘invisible
colleges’ is a term to say: “Okay, people
162
00:17:50,899 --> 00:17:56,490
are citing each other. They do not work
together in a co-working space, maybe, but
163
00:17:56,490 --> 00:18:00,820
they do research on the same topic.” And
that’s only plausible that they cite each
164
00:18:00,820 --> 00:18:06,540
other. And if we look at citation networks
and find people citing each other, that
165
00:18:06,540 --> 00:18:12,860
does not have necessarily to be something
bad. And we also have people who are
166
00:18:12,860 --> 00:18:18,519
concerned that there might be like
‘citation cartels’. So researchers citing
167
00:18:18,519 --> 00:18:27,270
each other not for purposes like the
research topics are closely connected, but
168
00:18:27,270 --> 00:18:35,720
to support each other in their career
prospects. And people do try to
169
00:18:35,720 --> 00:18:40,730
discriminate those invisible colleges from
citation cartels ex post from looking at
170
00:18:40,730 --> 00:18:45,990
metadata networks of publication and find
that a problem. And we have a discourse on
171
00:18:45,990 --> 00:18:58,299
that in the bibliometrics community. I
will show you some short quotes how people
172
00:18:58,299 --> 00:19:04,510
talk about those citation cartels. So e.g.
Davis in 2012 said: “George Franck warned
173
00:19:04,510 --> 00:19:08,960
us on the possibility of citation cartels
– groups of editors and journals working
174
00:19:08,960 --> 00:19:13,590
together for mutual benefit.” So we have
heard about their journal impact factors,
175
00:19:13,590 --> 00:19:23,020
so they... it’s believed that editors talk
to each other: “Hey you cite my journal,
176
00:19:23,020 --> 00:19:27,479
I cite your journal, and we both
will boost our impact factors.”
177
00:19:27,479 --> 00:19:32,899
So we have people trying
to detect those cartels,
178
00:19:32,899 --> 00:19:37,299
and Mongeon et al. wrote that:
“We have little knowledge
179
00:19:37,299 --> 00:19:40,639
about the phenomenon itself and
about where to draw the line between
180
00:19:40,639 --> 00:19:46,080
acceptable and unacceptable behavior.” So
we are having like moral discussions,
181
00:19:46,080 --> 00:19:53,560
about research ethics. And also we find
discussions about the fairness of the
182
00:19:53,560 --> 00:19:58,461
impact factors. So Yang et al. wrote:
“Disingenuously manipulating impact factor
183
00:19:58,461 --> 00:20:02,700
is the significant way to harm the
fairness of the impact factor.” And that’s
184
00:20:02,700 --> 00:20:09,780
a very interesting thing I think, because
why should an indicator be fair? So the...
185
00:20:09,780 --> 00:20:15,580
To believe that we have a fair measurement
of scientific quality relevance and rigor
186
00:20:15,580 --> 00:20:21,809
in one single like number, like their
journal impact factor, is not a small
187
00:20:21,809 --> 00:20:29,669
thing to say. And also we have a call for
detection and punishment. So Davis also
188
00:20:29,669 --> 00:20:33,780
wrote: “If disciplinary norms and decorum
cannot keep this kind of behavior at bay,
189
00:20:33,780 --> 00:20:39,700
the threat of being delisted from the JCR
may be necessary.” So we find the moral
190
00:20:39,700 --> 00:20:44,480
concerns on right and wrong. We find the
evocation of the fairness of indicators
191
00:20:44,480 --> 00:20:51,070
and we find the call for detection and
punishment. When I first heard about that
192
00:20:51,070 --> 00:20:56,640
phenomenon of citation cartels which is
believed to exist, I had something in mind
193
00:20:56,640 --> 00:21:03,740
which sounded... or it sounded like
familiar to me. Because we have a similar
194
00:21:03,740 --> 00:21:11,130
information retrieval discourse or a
discourse about ranking and power in a
195
00:21:11,130 --> 00:21:19,159
different area of society: in search
engine optimization. So I found a quote by
196
00:21:19,159 --> 00:21:26,830
Page et al., who developed the PageRank
algorithm – Google’s ranking algorithm –
197
00:21:26,830 --> 00:21:32,990
in 1999, which has changed since that a
lot. But they wrote also a paper about the
198
00:21:32,990 --> 00:21:42,500
social implications of the information
retrieval by the PageRank as an indicator.
199
00:21:42,500 --> 00:21:45,909
And wrote that: “These types of
personalized PageRanks are virtually
200
00:21:45,909 --> 00:21:50,150
immune to manipulation by commercial
interests. ... For example fast updating
201
00:21:50,150 --> 00:21:54,350
of documents is a very desirable feature,
but it is abused by people who want to
202
00:21:54,350 --> 00:22:00,529
manipulate the results of the search
engine.” And that was important to me to
203
00:22:00,529 --> 00:22:09,350
read because we also have like a narration
of abuse, of manipulation, the perception
204
00:22:09,350 --> 00:22:14,210
that that might be fair, so we have a fair
indicator and people try to betray it.
205
00:22:14,210 --> 00:22:21,790
And then we had in the early 2000s,
I recall having a private website
206
00:22:21,790 --> 00:22:24,560
with a public guest book and
getting link spam from people
207
00:22:24,560 --> 00:22:26,719
who wanted to boost their
Google PageRanks,
208
00:22:26,719 --> 00:22:32,779
and shortly afterwards Google
decided to punish link spam in their
209
00:22:32,779 --> 00:22:37,860
ranking algorithm. And then I got lots of
emails of people saying: “Please delete my
210
00:22:37,860 --> 00:22:43,890
post from your guestbook because Google’s
going to punish me for that.” We may say
211
00:22:43,890 --> 00:22:51,509
that this search engine optimization
discussion is now somehow settled and it’s
212
00:22:51,509 --> 00:22:57,940
accepted that Google's ranking is useful.
They have a secret algorithm, but it works
213
00:22:57,940 --> 00:23:05,250
and that is why it’s widely used. Although
that journal impact factor seems to be
214
00:23:05,250 --> 00:23:13,159
transparent it’s basically the same thing
that it's accepted to be useful and thus
215
00:23:13,159 --> 00:23:17,410
it's widely used. So the journal impact
factor, the SCI and the like. We have
216
00:23:17,410 --> 00:23:24,570
another analogy so that Google decides
which SEO behavior is regarded acceptable
217
00:23:24,570 --> 00:23:28,480
and punishes those who act against the
rules and thus holds an enormous amount of
218
00:23:28,480 --> 00:23:38,669
power, which has lots of implications and
led to the spreading of content management
219
00:23:38,669 --> 00:23:44,910
systems, for example, with search engine
optimization plugins etc. We also have
220
00:23:44,910 --> 00:23:52,809
this power concentration in the hands of
Clarivate (formerly ThomsonReuters) who
221
00:23:52,809 --> 00:23:59,059
host the database for the general impact
factor. And they decide on who’s going to
222
00:23:59,059 --> 00:24:05,140
be indexed in those journal citation
records and how is the algorithm, in
223
00:24:05,140 --> 00:24:12,200
detail, implemented in their databases. So
we have this power concentration there
224
00:24:12,200 --> 00:24:21,780
too, and I think if we think about this
analogy we might come to interesting
225
00:24:21,780 --> 00:24:29,770
thoughts but our time is running out so we
are going to give a take-home message.
226
00:24:29,770 --> 00:24:34,720
Tl;dr, we find that the scientific
community reacts with codes of conduct to
227
00:24:34,720 --> 00:24:39,929
a problem which is believed to exist. The
strategic citation – we have database
228
00:24:39,929 --> 00:24:44,830
providers which react with sanctions so
people are delisted from the journal
229
00:24:44,830 --> 00:24:50,240
citation records or journals are delisted
from the journal citation records to
230
00:24:50,240 --> 00:24:55,419
punish them for citation stacking. And we
have researchers and publishers who adapt
231
00:24:55,419 --> 00:25:05,159
their publication strategies in reaction
to this perceived algorithmic power. But
232
00:25:05,159 --> 00:25:12,470
if we want to understand this as a problem
we don’t have to only react to the
233
00:25:12,470 --> 00:25:19,179
algorithm but we have to address the power
structures. Who holds these instruments in
234
00:25:19,179 --> 00:25:24,210
in their hands? If we talk about
bibliometrics as an instrument and we
235
00:25:24,210 --> 00:25:27,830
should not only blame the algorithm – so
#dontblamethealgorithm.
236
00:25:27,830 --> 00:25:33,200
Thank you very much!
applause
237
00:25:37,979 --> 00:25:43,630
Herald: Thank you to Franziska, Teresa
and Judith, or in the reverse order.
238
00:25:44,510 --> 00:25:48,450
Thank you for shining a light on
how science is actually seen
239
00:25:48,450 --> 00:25:50,800
in its publications.
240
00:25:50,800 --> 00:25:52,410
As I started off as well,
241
00:25:52,410 --> 00:25:55,650
it’s more about
scratching each other a little bit.
242
00:25:55,650 --> 00:25:57,790
I have some questions here
from the audience.
243
00:25:57,790 --> 00:25:59,740
This is Microphone 2, please!
244
00:25:59,740 --> 00:26:05,319
Mic2: Yes, thank you for this interesting
talk. I have a question. You may be
245
00:26:05,319 --> 00:26:09,549
familiar with the term ‘measurement
dysfunction’, that if you provide a worker
246
00:26:09,549 --> 00:26:14,220
with an incentive to do a good job based
on some kind of metric then the worker
247
00:26:14,220 --> 00:26:19,960
will start optimizing for the metric
instead of trying to do a good job, and
248
00:26:19,960 --> 00:26:26,200
this is kind of inevitable. So, don’t you
see that maybe it could be treating the
249
00:26:26,200 --> 00:26:32,540
symptoms if we just react about code of
conduct, tweaking algorithms or addressing
250
00:26:32,540 --> 00:26:37,200
power structures. But instead we need to
remove the incentives that lead to this
251
00:26:37,200 --> 00:26:43,669
measurement dysfunction.
Judith: I would refer to this phenomenon
252
00:26:43,669 --> 00:26:50,740
as “perverse learning” – learning for the
grades you get but not for your intrinsic
253
00:26:50,740 --> 00:27:00,800
motivation to learn something. We observe
that in the science system. But if we only
254
00:27:00,800 --> 00:27:10,440
adapt the algorithm, so take away the
incentives, would be like you wouldn’t
255
00:27:10,440 --> 00:27:20,480
want to evaluate research at all which you
can probably want to do. But to whom would
256
00:27:20,480 --> 00:27:32,740
you address this call or this demand, so
“please do not have indicators” or… I give
257
00:27:32,740 --> 00:27:38,690
the question back to you. laughs
Herald: Okay, questions from the audience
258
00:27:38,690 --> 00:27:46,499
out there on the Internet, please. Your
mic is not working? Okay, then I go to
259
00:27:46,499 --> 00:27:52,130
Microphone 1, please Sir.
Mic1: Yeah, I want to have a provocative
260
00:27:52,130 --> 00:27:56,690
thesis. I think the fundamental problem is
not how these things are gamed but the
261
00:27:56,690 --> 00:28:01,200
fundamental problem is that if we think
the impact factor is a useful measurement
262
00:28:01,200 --> 00:28:04,599
for the quality of science.
Because I think it’s just not.
263
00:28:04,599 --> 00:28:06,950
applause
264
00:28:09,580 --> 00:28:12,110
Judith: Ahm.. I..
Mic 1: I guess that was obvious...
265
00:28:12,110 --> 00:28:13,490
Judith: Yeah, I would not say
266
00:28:13,490 --> 00:28:17,630
that the journal impact factor is
a measurement of scientific quality
267
00:28:17,630 --> 00:28:23,519
because no one has like
a definition of scientific quality.
268
00:28:23,519 --> 00:28:28,450
So what I can observe is only
people believe this journal impact factor
269
00:28:28,450 --> 00:28:36,649
to reflect some quality.
Maybe they are chasing a ghost but I…
270
00:28:36,649 --> 00:28:41,979
whether that’s a valid measure
is not so important to me,
271
00:28:41,979 --> 00:28:45,239
even if it were a relevant
or a valid measure,
272
00:28:45,239 --> 00:28:52,309
it would concern me
how it affects science.
273
00:28:52,989 --> 00:28:56,499
Herald: Okay, question from Microphone 3
there. Please.
274
00:28:56,499 --> 00:28:59,499
Mic3: Thanks for the interesting talk.
I have a question about
275
00:28:59,499 --> 00:29:04,400
the 5,000 authors paper.
Was that same paper published
276
00:29:04,400 --> 00:29:09,449
five thousand times or was it one paper
with ten page title page?
277
00:29:10,049 --> 00:29:15,259
Franziska: No, it was one paper ...
278
00:29:15,259 --> 00:29:19,599
... counting more than 7,000 words.
And the authorship,
279
00:29:19,599 --> 00:29:24,460
so authors and co-authors,
were more than 5,000.
280
00:29:24,460 --> 00:29:31,010
Mic3: Isn’t it obvious
that this is a fake?
281
00:29:31,010 --> 00:29:35,070
Franziska: Well that’s
what I meant earlier
282
00:29:35,070 --> 00:29:44,049
when saying, you have to see this within
its context. So physicists are working
283
00:29:44,049 --> 00:29:52,169
with this with Atlas, this detective
system. As there were some physicists in
284
00:29:52,169 --> 00:30:01,780
the audience they probably do know how
this works. I do not. But as they claim
285
00:30:01,780 --> 00:30:08,330
it’s so much work to work with this, and
it, as I said, requires so high
286
00:30:08,330 --> 00:30:19,120
maintenance it’s... They obviously have
yeah...
287
00:30:19,120 --> 00:30:21,719
Mic3: So everybody who contributed was
listed?
288
00:30:21,719 --> 00:30:28,809
Judith: Exactly, that’s it. And if this is
ethically correct or not, well, this is
289
00:30:28,809 --> 00:30:33,820
something which needs to be discussed,
right? This is why we have this talk, as
290
00:30:33,820 --> 00:30:39,910
we want to make this transparent, and
contribute it to an open discussion.
291
00:30:39,910 --> 00:30:44,610
Herald: Okay, I’m sorry guys. I have to
cut off here because our emission out
292
00:30:44,610 --> 00:30:49,229
there in space is coming to an end.
I suggest that you guys
293
00:30:49,229 --> 00:30:52,879
find each other somewhere,
maybe in the tea house or...
294
00:30:52,879 --> 00:30:55,019
Judith: Sure. We are around, we are here.
295
00:30:55,019 --> 00:30:58,149
Herald: You are around. I would love to
have lots of applause for these ladies,
296
00:30:58,149 --> 00:31:03,030
for it really lights on
how these algorithms
297
00:31:03,030 --> 00:31:05,299
not or are working. Thank you very much!
298
00:31:05,299 --> 00:31:06,639
Judith: Thank you!
299
00:31:06,639 --> 00:31:21,629
postroll music
300
00:31:21,629 --> 00:31:26,741
subtitles created by c3subtitles.de
in the year 2018