WEBVTT
00:00:00.000 --> 00:00:14.595
34C3 preroll music
00:00:14.595 --> 00:00:17.870
Herald: Please give a warm welcome here.
00:00:17.870 --> 00:00:23.610
It’s Franziska, Teresa, and Judith.
00:00:23.610 --> 00:00:26.510
Judith, you have the stage, thank you.
00:00:26.510 --> 00:00:28.470
Judith Hartstein: Thank you, thanks!
00:00:28.470 --> 00:00:33.710
applause
00:00:33.710 --> 00:00:57.430
inaudible
00:00:57.430 --> 00:01:00.219
Judith: We believe that
scientific performance indicators
00:01:00.219 --> 00:01:03.269
are widely applied to inform
funding decisions and to
00:01:03.269 --> 00:01:08.270
determine the availability of career
opportunities. So, those of you who are
00:01:08.270 --> 00:01:14.420
working in science or have had a look into
the science system might agree to that.
00:01:14.420 --> 00:01:18.840
And we want to understand evaluative
bibliometrics as algorithmic science
00:01:18.840 --> 00:01:27.640
evaluation instruments to highlight some
things that do occur also with other
00:01:27.640 --> 00:01:38.020
algorithmic instruments of evaluation. And
so we’re going to start with a quote from
00:01:38.020 --> 00:01:44.679
a publication in 2015 which reads “As the
tyranny of bibliometrics tightens its
00:01:44.679 --> 00:01:49.130
grip, it is having a disastrous effect on
the model of science presented to young
00:01:49.130 --> 00:01:58.999
researchers.” We have heard the talk of
hanno already, and he’s basically also
00:01:58.999 --> 00:02:06.679
talking about problems in the science
system and the reputation by the
00:02:06.679 --> 00:02:14.170
indicators. And the question is, is
bibliometrics the bad guy here? If you
00:02:14.170 --> 00:02:18.530
speak of ‘tyranny of bibliometrics’, who
is the actor doing this? Or are maybe
00:02:18.530 --> 00:02:24.950
bibliometricians the problem? We want to
contextualize our talk into the growing
00:02:24.950 --> 00:02:30.440
movement of Reflexive Metrics. So those
who are doing science studies, social
00:02:30.440 --> 00:02:34.940
studies of science, scientometrics and
bibliometrics. The movement of Reflexive
00:02:34.940 --> 00:02:41.690
Metrics. So the basic idea is to say:
“Okay, we have to accept accountability if
00:02:41.690 --> 00:02:45.540
we do bibliometrics and scientometrics.”
We have to understand the effects of
00:02:45.540 --> 00:02:54.220
algorithmic evaluation on science, and we
will try not to be the bad guy. And the
NOTE Paragraph
00:02:54.220 --> 00:03:03.640
main mediator of the science evaluation
which is perceived by the researchers is
00:03:03.640 --> 00:03:09.630
the algorithm. I will hand over the
microphone to… or I will not hand over the
00:03:09.630 --> 00:03:14.440
microphone but I will hand over the talk
to Teresa. She’s going to talk about
00:03:14.440 --> 00:03:19.690
"Datafication of Scientific Evaluation".
00:03:19.690 --> 00:03:24.180
Teresa Isigkeit: Okay. I hope you can
hear me. No? Yes? Okay.
00:03:24.180 --> 00:03:25.890
Judith: mumbling
00:03:25.890 --> 00:03:29.480
When we think about the science system
what do we expect?
00:03:29.480 --> 00:03:33.700
What can society expect
from a scientific system?
00:03:33.700 --> 00:03:38.200
In general, we would say
reliable and truthful knowledge,
00:03:38.200 --> 00:03:41.810
that is scrutinized by
the scientific community.
00:03:41.810 --> 00:03:44.320
So where can we find this knowledge?
00:03:44.320 --> 00:03:47.140
Normally in publications.
00:03:47.140 --> 00:03:51.920
So with these publications,
can we actually say
00:03:51.920 --> 00:03:58.670
whether science is bad or good? Or is
there better science than others?
00:03:58.670 --> 00:04:03.350
In the era of
digital publication databases,
00:04:03.350 --> 00:04:07.280
there’s big datasets of publications.
00:04:07.280 --> 00:04:12.330
And these are used to
evaluate and calculate
00:04:12.330 --> 00:04:16.840
the quality of scientific output.
00:04:16.840 --> 00:04:22.720
So in general, with this metadata
we can tell you
00:04:22.720 --> 00:04:26.400
who is the author of a publication,
00:04:26.400 --> 00:04:30.020
where is the home institution
of this author,
00:04:30.020 --> 00:04:37.960
or which types of citations are in
the bibliographic information.
00:04:37.960 --> 00:04:44.890
This is used in the calculation
of bibliometric indicators.
00:04:44.890 --> 00:04:51.580
For example if you take the
journal impact factors,
00:04:51.580 --> 00:04:57.650
which is a citation based indicator,
you can compare different journals.
00:04:57.650 --> 00:05:03.750
And maybe say which journals
are performing better than others
00:05:03.750 --> 00:05:09.360
or if the journal factor has increased or
decreased over the years.
00:05:09.360 --> 00:05:15.330
Another example would be the
Hirsch-Index for individual scientists,
00:05:15.330 --> 00:05:22.670
which is also widely used when
scientists apply for jobs. So they put
00:05:22.670 --> 00:05:27.710
these numbers in their CVs and supposedly
this tells you something about the quality
00:05:27.710 --> 00:05:36.270
of research those scientists are
conducting. With the availability of the
00:05:36.270 --> 00:05:45.590
data we can see an increase in its usage.
And in a scientific environment in which
00:05:45.590 --> 00:05:52.260
data-driven science is established,
scientific conduct decisions regarding
00:05:52.260 --> 00:06:04.419
hiring or funding heavily rely on these
indicators. There’s maybe a naive belief
00:06:04.419 --> 00:06:12.020
that these indicators that are data-driven
and rely on data that is collected in the
00:06:12.020 --> 00:06:26.610
database is a more objective metric that
we can use. So here's a quote by Rieder
00:06:26.610 --> 00:06:32.430
and Simon: “In this brave new world trust
no longer resides in the integrity of
00:06:32.430 --> 00:06:38.710
individual truth-tellers or the veracity
of prestigious institutions, but is placed
00:06:38.710 --> 00:06:44.480
in highly formalized procedures enacted
through disciplined self-restraint.
00:06:44.480 --> 00:06:53.150
Numbers cease to be supplements.” So we
see a change of an evaluation system that
00:06:53.150 --> 00:07:00.449
is relying on expert knowledge to a system
of algorithmic science evaluation. In this
00:07:00.449 --> 00:07:05.699
change there’s a belief in a
depersonalization of the system and the
00:07:05.699 --> 00:07:15.090
perception of algorithms as the rule of
law. So when looking at the interaction
00:07:15.090 --> 00:07:26.010
between the algorithm and scientists we
can tell that this relationship is not as
00:07:26.010 --> 00:07:35.169
easy as it seems. Algorithms are not in
fact objective. They carry social meaning
00:07:35.169 --> 00:07:43.020
and human agency. They are used to
construct a reality and algorithms don’t
00:07:43.020 --> 00:07:48.290
come naturally. They don’t grow on trees
and can be picked by scientists and people
00:07:48.290 --> 00:07:55.120
who evaluate the scientific system, so we
have to be reflective and think about
00:07:55.120 --> 00:08:04.910
which social meanings the algorithm holds.
So when there is a code that the algorithm
00:08:04.910 --> 00:08:11.169
uses, there is a subjective meaning in
this code, and there is agency in this
00:08:11.169 --> 00:08:17.370
code, and you can’t just say, oh, this is
a perfect construction of the reality of
00:08:17.370 --> 00:08:22.350
scientific system. So the belief that this
tells you more about the quality of
00:08:22.350 --> 00:08:31.820
research is not a good indicator. So when
you think about the example of citation
00:08:31.820 --> 00:08:37.049
counts the algorithm reads the
bibliographic information of a publication
00:08:37.049 --> 00:08:47.230
from the database. So scientists, they
cite papers that relate to their studies.
00:08:47.230 --> 00:08:55.740
But we don’t actually know which of these
citations are more meaningful than others,
00:08:55.740 --> 00:09:01.319
so they’re not as easily comparable. But
the algorithms give you the belief they
00:09:01.319 --> 00:09:11.230
are, so relevance is not as easily put
into an algorithm and there is different
00:09:11.230 --> 00:09:19.199
types of citations. So the scientists
perceive this use of the algorithms also
00:09:19.199 --> 00:09:24.790
as a powerful instrument. And so the
algorithm has some sway above the
00:09:24.790 --> 00:09:30.019
scientists because they rely so much on
those indicators to further their careers,
00:09:30.019 --> 00:09:38.069
to get a promotion, or get funding for
their next research projects. So we have a
00:09:38.069 --> 00:09:42.970
reciprocal relationship between the
algorithm and the scientists, and this
00:09:42.970 --> 00:09:52.399
creates a new construction of reality. So
we can conclude that governance by
00:09:52.399 --> 00:09:59.019
algorithms leads to behavioral adaptation
in scientists, and one of these examples
00:09:59.019 --> 00:10:08.069
that uses the Science Citation Index will
be given from Franziska.
00:10:08.069 --> 00:10:12.610
Franziska Sörgel: Thanks for the
handover! Yes, let me start.
00:10:12.610 --> 00:10:16.070
I’m focusing on reputation
and authorship as you can see
00:10:16.070 --> 00:10:21.420
on the slide, and first let me
start with a quote
00:10:21.420 --> 00:10:27.490
by Eugene Garfield, which says: “Is it
reasonable to assume that if I cite a
00:10:27.490 --> 00:10:32.889
paper that I would probably be interested
in those papers which subsequently cite it
00:10:32.889 --> 00:10:38.630
as well as my own paper. Indeed, I have
observed on several occasions that people
00:10:38.630 --> 00:10:45.399
preferred to cite the articles I had cited
rather than cite me! It would seem to me
00:10:45.399 --> 00:10:51.321
that this is the basis for the building up
of the ‘logical network’ for the Citation
00:10:51.321 --> 00:11:01.709
Index service.” So, actually, this Science
Citation Index which is described here was
00:11:01.709 --> 00:11:07.970
mainly developed in order to solve the
problems of information retrieval. Eugene
00:11:07.970 --> 00:11:16.259
Garfield, also founder of this Science
Citation Index – short: SCI – noted or
00:11:16.259 --> 00:11:22.170
began to note a huge interest in
reciprocal publication behavior. He
00:11:22.170 --> 00:11:27.470
recognized the increasing interest as a
strategic instrument to exploit
00:11:27.470 --> 00:11:33.410
intellectual property. And indeed, the
interest in the SCI – and its data –
00:11:33.410 --> 00:11:39.189
successively became more relevant within
the disciplines, and its usage extended.
00:11:39.189 --> 00:11:45.800
Later, [Derek J.] de Solla Price, another
social scientist, asked or claimed for a
00:11:45.800 --> 00:11:53.029
better research on the topic, as it
currently also meant a crisis in science,
NOTE Paragraph
00:11:53.029 --> 00:11:59.329
and stated: “If a paper was cited once,
it would get cited again and
00:11:59.329 --> 00:12:05.089
again, so the main problem was that the
rich would get richer”, which is also
00:12:05.089 --> 00:12:12.029
known as the “Matthew Effect”. Finally,
the SCI and its use turned into a system
00:12:12.029 --> 00:12:17.860
which was and still is used as a
reciprocal citation system, and became a
00:12:17.860 --> 00:12:25.119
central and global actor. Once a paper was
cited, the probability it was cited again
00:12:25.119 --> 00:12:30.720
was higher, and it would even extend its
own influence on a certain topic within
00:12:30.720 --> 00:12:38.369
the scientific field. So it was known that
you would either read a certain article
00:12:38.369 --> 00:12:48.999
and people would do research on a certain
topic or subject. So this phenomenon would
NOTE Paragraph
00:12:48.999 --> 00:12:58.569
rise to an instrument of disciplining
science and created power structures.
00:12:58.569 --> 00:13:04.530
Let me show you one example which is
closely connected to this phenomenon
00:13:04.530 --> 00:13:10.920
I just told you about – and I don’t know
if here in this room there are any
00:13:10.920 --> 00:13:18.680
astronomers or physicists?
Yeah, there are few, okay.
00:13:18.680 --> 00:13:24.580
That’s great, actually.
So in the next slide, here,
00:13:24.580 --> 00:13:32.990
we have a table with a time
window from 2010 to 2016, and social
00:13:32.990 --> 00:13:41.069
scientists from Berlin found out that the
co-authorship within the field of physics
00:13:41.069 --> 00:13:50.519
extended by 58 on a yearly basis in this
time window. So this is actually already
00:13:50.519 --> 00:13:55.920
very high, but they also found another
very extreme case. They found one paper
00:13:55.920 --> 00:14:07.399
which had roundabout 7,000 words and the
mentioned authorship of 5,000. So, in
00:14:07.399 --> 00:14:15.389
average, the contribution of each
scientist or researcher of this paper who
00:14:15.389 --> 00:14:28.910
was mentioned was 1.1 word. Sounds
strange, yeah. And so of course you have
00:14:28.910 --> 00:14:34.519
to see this in a certain context, and
maybe we can talk about this later on,
00:14:34.519 --> 00:14:41.380
because it has to do with Atlas particle
detector, which requires high maintenance
00:14:41.380 --> 00:14:46.019
and stuff. But still, the number of
authorship, and you can see this
00:14:46.019 --> 00:14:52.519
regardless which scientific field we are
talking about, generally increased the
00:14:52.519 --> 00:15:05.360
last years. It remains a problem
especially for the reputation, obviously.
00:15:05.360 --> 00:15:12.399
It remains a problem that there is such
high pressure on nowadays researchers.
00:15:12.399 --> 00:15:20.200
Still, of course, we have ethics and
research requires standards of
00:15:20.200 --> 00:15:25.689
responsibility. And for example there’s
one, there’s other ones, but there’s one
00:15:25.689 --> 00:15:30.850
here on the slide: the “Australian Code
for the Responsible Conduct of Research”
00:15:30.850 --> 00:15:36.689
which says: “The right to authorship is
not tied to position or profession and
00:15:36.689 --> 00:15:41.060
does not depend on whether the
contribution was paid for or voluntary.
00:15:41.060 --> 00:15:46.409
It is not enough to have provided
materials or routine technical support,
00:15:46.419 --> 00:15:50.969
or to have made the measurements
on which the publication is based.
00:15:50.969 --> 00:15:55.129
Substantial intellectual involvement
is required.”
00:15:55.129 --> 00:16:03.259
So yeah, this is, could be one rule
to work with or to work by, to follow.
00:16:03.259 --> 00:16:08.079
And still we have this problem
of reputation which remains,
00:16:08.079 --> 00:16:11.480
and where I hand over to Judith again.
00:16:11.480 --> 00:16:19.739
Judith: Thank you. So we’re going to speak
about strategic citation now. So if you
00:16:19.739 --> 00:16:29.540
put this point of reputation like that,
you may say: So the researcher does find
00:16:29.540 --> 00:16:36.239
something in his research, his or her
research, and addresses the publication
00:16:36.239 --> 00:16:40.049
describing it to the community. And the
community, the scientific community
00:16:40.049 --> 00:16:46.269
rewards the researcher with reputation.
And now the algorithm, which is like
00:16:46.269 --> 00:16:54.730
perceived to be a new thing, is mediating
the visibility of the researcher’s results
00:16:54.730 --> 00:17:00.720
to the community, and is also mediating
the rewards – the career opportunities or
00:17:00.720 --> 00:17:04.919
the funding decisions etc. And what
happens now and what is plausible to
00:17:04.919 --> 00:17:10.398
happen is that the researcher addresses
his or her research also to the algorithm
00:17:10.398 --> 00:17:21.039
in terms of citing those who are evaluated
by the algorithm, who he wants to support,
00:17:21.039 --> 00:17:29.230
and also in terms of strategic keywording
etc. And that’s the only thing which
00:17:29.230 --> 00:17:33.809
happens new, might be a perspective on
that. So the one thing new: the algorithm
00:17:33.809 --> 00:17:40.700
is addressed as a recipient of scientific
publications. And it is like far-fetched
00:17:40.700 --> 00:17:46.260
to discriminate between so-called and
‘visible colleges’ and ‘citation cartels’.
00:17:46.260 --> 00:17:50.899
What do I mean by that? So ‘invisible
colleges’ is a term to say: “Okay, people
00:17:50.899 --> 00:17:56.490
are citing each other. They do not work
together in a co-working space, maybe, but
00:17:56.490 --> 00:18:00.820
they do research on the same topic.” And
that’s only plausible that they cite each
00:18:00.820 --> 00:18:06.540
other. And if we look at citation networks
and find people citing each other, that
00:18:06.540 --> 00:18:12.860
does not have necessarily to be something
bad. And we also have people who are
00:18:12.860 --> 00:18:18.519
concerned that there might be like
‘citation cartels’. So researchers citing
00:18:18.519 --> 00:18:27.270
each other not for purposes like the
research topics are closely connected, but
00:18:27.270 --> 00:18:35.720
to support each other in their career
prospects. And people do try to
00:18:35.720 --> 00:18:40.730
discriminate those invisible colleges from
citation cartels ex post from looking at
00:18:40.730 --> 00:18:45.990
metadata networks of publication and find
that a problem. And we have a discourse on
00:18:45.990 --> 00:18:58.299
that in the bibliometrics community. I
will show you some short quotes how people
00:18:58.299 --> 00:19:04.510
talk about those citation cartels. So e.g.
Davis in 2012 said: “George Franck warned
00:19:04.510 --> 00:19:08.960
us on the possibility of citation cartels
– groups of editors and journals working
00:19:08.960 --> 00:19:13.590
together for mutual benefit.” So we have
heard about their journal impact factors,
00:19:13.590 --> 00:19:23.020
so they... it’s believed that editors talk
to each other: “Hey you cite my journal,
00:19:23.020 --> 00:19:27.479
I cite your journal, and we both
will boost our impact factors.”
00:19:27.479 --> 00:19:32.899
So we have people trying
to detect those cartels,
00:19:32.899 --> 00:19:37.299
and Mongeon et al. wrote that:
“We have little knowledge
00:19:37.299 --> 00:19:40.639
about the phenomenon itself and
about where to draw the line between
00:19:40.639 --> 00:19:46.080
acceptable and unacceptable behavior.” So
we are having like moral discussions,
00:19:46.080 --> 00:19:53.560
about research ethics. And also we find
discussions about the fairness of the
00:19:53.560 --> 00:19:58.461
impact factors. So Yang et al. wrote:
“Disingenuously manipulating impact factor
00:19:58.461 --> 00:20:02.700
is the significant way to harm the
fairness of the impact factor.” And that’s
00:20:02.700 --> 00:20:09.780
a very interesting thing I think, because
why should an indicator be fair? So the...
00:20:09.780 --> 00:20:15.580
To believe that we have a fair measurement
of scientific quality relevance and rigor
00:20:15.580 --> 00:20:21.809
in one single like number, like their
journal impact factor, is not a small
00:20:21.809 --> 00:20:29.669
thing to say. And also we have a call for
detection and punishment. So Davis also
00:20:29.669 --> 00:20:33.780
wrote: “If disciplinary norms and decorum
cannot keep this kind of behavior at bay,
00:20:33.780 --> 00:20:39.700
the threat of being delisted from the JCR
may be necessary.” So we find the moral
00:20:39.700 --> 00:20:44.480
concerns on right and wrong. We find the
evocation of the fairness of indicators
00:20:44.480 --> 00:20:51.070
and we find the call for detection and
punishment. When I first heard about that
00:20:51.070 --> 00:20:56.640
phenomenon of citation cartels which is
believed to exist, I had something in mind
00:20:56.640 --> 00:21:03.740
which sounded... or it sounded like
familiar to me. Because we have a similar
00:21:03.740 --> 00:21:11.130
information retrieval discourse or a
discourse about ranking and power in a
00:21:11.130 --> 00:21:19.159
different area of society: in search
engine optimization. So I found a quote by
00:21:19.159 --> 00:21:26.830
Page et al., who developed the PageRank
algorithm – Google’s ranking algorithm –
00:21:26.830 --> 00:21:32.990
in 1999, which has changed since that a
lot. But they wrote also a paper about the
00:21:32.990 --> 00:21:42.500
social implications of the information
retrieval by the PageRank as an indicator.
00:21:42.500 --> 00:21:45.909
And wrote that: “These types of
personalized PageRanks are virtually
00:21:45.909 --> 00:21:50.150
immune to manipulation by commercial
interests. ... For example fast updating
00:21:50.150 --> 00:21:54.350
of documents is a very desirable feature,
but it is abused by people who want to
00:21:54.350 --> 00:22:00.529
manipulate the results of the search
engine.” And that was important to me to
00:22:00.529 --> 00:22:09.350
read because we also have like a narration
of abuse, of manipulation, the perception
00:22:09.350 --> 00:22:14.210
that that might be fair, so we have a fair
indicator and people try to betray it.
00:22:14.210 --> 00:22:21.790
And then we had in the early 2000s,
I recall having a private website
00:22:21.790 --> 00:22:24.560
with a public guest book and
getting link spam from people
00:22:24.560 --> 00:22:26.719
who wanted to boost their
Google PageRanks,
00:22:26.719 --> 00:22:32.779
and shortly afterwards Google
decided to punish link spam in their
00:22:32.779 --> 00:22:37.860
ranking algorithm. And then I got lots of
emails of people saying: “Please delete my
00:22:37.860 --> 00:22:43.890
post from your guestbook because Google’s
going to punish me for that.” We may say
00:22:43.890 --> 00:22:51.509
that this search engine optimization
discussion is now somehow settled and it’s
00:22:51.509 --> 00:22:57.940
accepted that Google's ranking is useful.
They have a secret algorithm, but it works
00:22:57.940 --> 00:23:05.250
and that is why it’s widely used. Although
that journal impact factor seems to be
00:23:05.250 --> 00:23:13.159
transparent it’s basically the same thing
that it's accepted to be useful and thus
00:23:13.159 --> 00:23:17.410
it's widely used. So the journal impact
factor, the SCI and the like. We have
00:23:17.410 --> 00:23:24.570
another analogy so that Google decides
which SEO behavior is regarded acceptable
00:23:24.570 --> 00:23:28.480
and punishes those who act against the
rules and thus holds an enormous amount of
00:23:28.480 --> 00:23:38.669
power, which has lots of implications and
led to the spreading of content management
00:23:38.669 --> 00:23:44.910
systems, for example, with search engine
optimization plugins etc. We also have
00:23:44.910 --> 00:23:52.809
this power concentration in the hands of
Clarivate (formerly ThomsonReuters) who
00:23:52.809 --> 00:23:59.059
host the database for the general impact
factor. And they decide on who’s going to
00:23:59.059 --> 00:24:05.140
be indexed in those journal citation
records and how is the algorithm, in
00:24:05.140 --> 00:24:12.200
detail, implemented in their databases. So
we have this power concentration there
00:24:12.200 --> 00:24:21.780
too, and I think if we think about this
analogy we might come to interesting
00:24:21.780 --> 00:24:29.770
thoughts but our time is running out so we
are going to give a take-home message.
00:24:29.770 --> 00:24:34.720
Tl;dr, we find that the scientific
community reacts with codes of conduct to
00:24:34.720 --> 00:24:39.929
a problem which is believed to exist. The
strategic citation – we have database
00:24:39.929 --> 00:24:44.830
providers which react with sanctions so
people are delisted from the journal
00:24:44.830 --> 00:24:50.240
citation records or journals are delisted
from the journal citation records to
00:24:50.240 --> 00:24:55.419
punish them for citation stacking. And we
have researchers and publishers who adapt
00:24:55.419 --> 00:25:05.159
their publication strategies in reaction
to this perceived algorithmic power. But
00:25:05.159 --> 00:25:12.470
if we want to understand this as a problem
we don’t have to only react to the
00:25:12.470 --> 00:25:19.179
algorithm but we have to address the power
structures. Who holds these instruments in
00:25:19.179 --> 00:25:24.210
in their hands? If we talk about
bibliometrics as an instrument and we
00:25:24.210 --> 00:25:27.830
should not only blame the algorithm – so
#dontblamethealgorithm.
00:25:27.830 --> 00:25:33.200
Thank you very much!
applause
00:25:37.979 --> 00:25:43.630
Herald: Thank you to Franziska, Teresa
and Judith, or in the reverse order.
00:25:44.510 --> 00:25:48.450
Thank you for shining a light on
how science is actually seen
00:25:48.450 --> 00:25:50.800
in its publications.
00:25:50.800 --> 00:25:52.410
As I started off as well,
00:25:52.410 --> 00:25:55.650
it’s more about
scratching each other a little bit.
00:25:55.650 --> 00:25:57.790
I have some questions here
from the audience.
00:25:57.790 --> 00:25:59.740
This is Microphone 2, please!
00:25:59.740 --> 00:26:05.319
Mic2: Yes, thank you for this interesting
talk. I have a question. You may be
00:26:05.319 --> 00:26:09.549
familiar with the term ‘measurement
dysfunction’, that if you provide a worker
00:26:09.549 --> 00:26:14.220
with an incentive to do a good job based
on some kind of metric then the worker
00:26:14.220 --> 00:26:19.960
will start optimizing for the metric
instead of trying to do a good job, and
00:26:19.960 --> 00:26:26.200
this is kind of inevitable. So, don’t you
see that maybe it could be treating the
00:26:26.200 --> 00:26:32.540
symptoms if we just react about code of
conduct, tweaking algorithms or addressing
00:26:32.540 --> 00:26:37.200
power structures. But instead we need to
remove the incentives that lead to this
00:26:37.200 --> 00:26:43.669
measurement dysfunction.
Judith: I would refer to this phenomenon
00:26:43.669 --> 00:26:50.740
as “perverse learning” – learning for the
grades you get but not for your intrinsic
00:26:50.740 --> 00:27:00.800
motivation to learn something. We observe
that in the science system. But if we only
00:27:00.800 --> 00:27:10.440
adapt the algorithm, so take away the
incentives, would be like you wouldn’t
00:27:10.440 --> 00:27:20.480
want to evaluate research at all which you
can probably want to do. But to whom would
00:27:20.480 --> 00:27:32.740
you address this call or this demand, so
“please do not have indicators” or… I give
00:27:32.740 --> 00:27:38.690
the question back to you. laughs
Herald: Okay, questions from the audience
00:27:38.690 --> 00:27:46.499
out there on the Internet, please. Your
mic is not working? Okay, then I go to
00:27:46.499 --> 00:27:52.130
Microphone 1, please Sir.
Mic1: Yeah, I want to have a provocative
00:27:52.130 --> 00:27:56.690
thesis. I think the fundamental problem is
not how these things are gamed but the
00:27:56.690 --> 00:28:01.200
fundamental problem is that if we think
the impact factor is a useful measurement
00:28:01.200 --> 00:28:04.599
for the quality of science.
Because I think it’s just not.
00:28:04.599 --> 00:28:06.950
applause
00:28:09.580 --> 00:28:12.110
Judith: Ahm.. I..
Mic 1: I guess that was obvious...
00:28:12.110 --> 00:28:13.490
Judith: Yeah, I would not say
00:28:13.490 --> 00:28:17.630
that the journal impact factor is
a measurement of scientific quality
00:28:17.630 --> 00:28:23.519
because no one has like
a definition of scientific quality.
00:28:23.519 --> 00:28:28.450
So what I can observe is only
people believe this journal impact factor
00:28:28.450 --> 00:28:36.649
to reflect some quality.
Maybe they are chasing a ghost but I…
00:28:36.649 --> 00:28:41.979
whether that’s a valid measure
is not so important to me,
00:28:41.979 --> 00:28:45.239
even if it were a relevant
or a valid measure,
00:28:45.239 --> 00:28:52.309
it would concern me
how it affects science.
00:28:52.989 --> 00:28:56.499
Herald: Okay, question from Microphone 3
there. Please.
00:28:56.499 --> 00:28:59.499
Mic3: Thanks for the interesting talk.
I have a question about
00:28:59.499 --> 00:29:04.400
the 5,000 authors paper.
Was that same paper published
00:29:04.400 --> 00:29:09.449
five thousand times or was it one paper
with ten page title page?
00:29:10.049 --> 00:29:15.259
Franziska: No, it was one paper ...
00:29:15.259 --> 00:29:19.599
... counting more than 7,000 words.
And the authorship,
00:29:19.599 --> 00:29:24.460
so authors and co-authors,
were more than 5,000.
00:29:24.460 --> 00:29:31.010
Mic3: Isn’t it obvious
that this is a fake?
00:29:31.010 --> 00:29:35.070
Franziska: Well that’s
what I meant earlier
00:29:35.070 --> 00:29:44.049
when saying, you have to see this within
its context. So physicists are working
00:29:44.049 --> 00:29:52.169
with this with Atlas, this detective
system. As there were some physicists in
00:29:52.169 --> 00:30:01.780
the audience they probably do know how
this works. I do not. But as they claim
00:30:01.780 --> 00:30:08.330
it’s so much work to work with this, and
it, as I said, requires so high
00:30:08.330 --> 00:30:19.120
maintenance it’s... They obviously have
yeah...
00:30:19.120 --> 00:30:21.719
Mic3: So everybody who contributed was
listed?
00:30:21.719 --> 00:30:28.809
Judith: Exactly, that’s it. And if this is
ethically correct or not, well, this is
00:30:28.809 --> 00:30:33.820
something which needs to be discussed,
right? This is why we have this talk, as
00:30:33.820 --> 00:30:39.910
we want to make this transparent, and
contribute it to an open discussion.
00:30:39.910 --> 00:30:44.610
Herald: Okay, I’m sorry guys. I have to
cut off here because our emission out
00:30:44.610 --> 00:30:49.229
there in space is coming to an end.
I suggest that you guys
00:30:49.229 --> 00:30:52.879
find each other somewhere,
maybe in the tea house or...
00:30:52.879 --> 00:30:55.019
Judith: Sure. We are around, we are here.
00:30:55.019 --> 00:30:58.149
Herald: You are around. I would love to
have lots of applause for these ladies,
00:30:58.149 --> 00:31:03.030
for it really lights on
how these algorithms
00:31:03.030 --> 00:31:05.299
not or are working. Thank you very much!
00:31:05.299 --> 00:31:06.639
Judith: Thank you!
00:31:06.639 --> 00:31:21.629
postroll music
00:31:21.629 --> 00:31:26.741
subtitles created by c3subtitles.de
in the year 2018