34C3 preroll music
Herald: Please give a warm welcome here.
It’s Franziska, Teresa, and Judith.
Judith, you have the stage, thank you.
Judith Hartstein: Thank you, thanks!
applause
inaudible
Judith: We believe that scientific performance indicators
are widely applied to inform funding decisions and to
determine the availability of career opportunities. So, those of you who are
working in science or have had a look into the science system might agree to that.
And we want to understand evaluative bibliometrics as algorithmic science
evaluation instruments to highlight some things that do occur also with other
algorithmic instruments of evaluation. And so we’re going to start with a quote from
a publication in 2015 which reads “As the tyranny of bibliometrics tightens its
grip, it is having a disastrous effect on the model of science presented to young
researchers.” We have heard the talk of hanno already, and he’s basically also
talking about problems in the science system and the reputation by the
indicators. And the question is, is bibliometrics the bad guy here? If you
speak of ‘tyranny of bibliometrics’, who is the actor doing this? Or are maybe
bibliometricians the problem? We want to contextualize our talk into the growing
movement of Reflexive Metrics. So those who are doing science studies, social
studies of science, scientometrics and bibliometrics. The movement of Reflexive
Metrics. So the basic idea is to say: “Okay, we have to accept accountability if
we do bibliometrics and scientometrics.” We have to understand the effects of
algorithmic evaluation on science, and we will try not to be the bad guy. And the
main mediator of the science evaluation which is perceived by the researchers is
the algorithm. I will hand over the microphone to… or I will not hand over the
microphone but I will hand over the talk to Teresa. She’s going to talk about
"Datafication of Scientific Evaluation".
Teresa Isigkeit: Okay. I hope you can hear me. No? Yes? Okay.
Judith: mumbling
When we think about the science system what do we expect?
What can society expect from a scientific system?
In general, we would say reliable and truthful knowledge,
that is scrutinized by the scientific community.
So where can we find this knowledge?
Normally in publications.
So with these publications, can we actually say
whether science is bad or good? Or is there better science than others?
In the era of digital publication databases,
there’s big datasets of publications.
And these are used to evaluate and calculate
the quality of scientific output.
So in general, with this metadata we can tell you
who is the author of a publication,
where is the home institution of this author,
or which types of citations are in the bibliographic information.
This is used in the calculation of bibliometric indicators.
For example if you take the journal impact factors,
which is a citation based indicator, you can compare different journals.
And maybe say which journals are performing better than others
or if the journal factor has increased or decreased over the years.
Another example would be the Hirsch-Index for individual scientists,
which is also widely used when scientists apply for jobs. So they put
these numbers in their CVs and supposedly this tells you something about the quality
of research those scientists are conducting. With the availability of the
data we can see an increase in its usage. And in a scientific environment in which
data-driven science is established, scientific conduct decisions regarding
hiring or funding heavily rely on these indicators. There’s maybe a naive belief
that these indicators that are data-driven and rely on data that is collected in the
database is a more objective metric that we can use. So here's a quote by Rieder
and Simon: “In this brave new world trust no longer resides in the integrity of
individual truth-tellers or the veracity of prestigious institutions, but is placed
in highly formalized procedures enacted through disciplined self-restraint.
Numbers cease to be supplements.” So we see a change of an evaluation system that
is relying on expert knowledge to a system of algorithmic science evaluation. In this
change there’s a belief in a depersonalization of the system and the
perception of algorithms as the rule of law. So when looking at the interaction
between the algorithm and scientists we can tell that this relationship is not as
easy as it seems. Algorithms are not in fact objective. They carry social meaning
and human agency. They are used to construct a reality and algorithms don’t
come naturally. They don’t grow on trees and can be picked by scientists and people
who evaluate the scientific system, so we have to be reflective and think about
which social meanings the algorithm holds. So when there is a code that the algorithm
uses, there is a subjective meaning in this code, and there is agency in this
code, and you can’t just say, oh, this is a perfect construction of the reality of
scientific system. So the belief that this tells you more about the quality of
research is not a good indicator. So when you think about the example of citation
counts the algorithm reads the bibliographic information of a publication
from the database. So scientists, they cite papers that relate to their studies.
But we don’t actually know which of these citations are more meaningful than others,
so they’re not as easily comparable. But the algorithms give you the belief they
are, so relevance is not as easily put into an algorithm and there is different
types of citations. So the scientists perceive this use of the algorithms also
as a powerful instrument. And so the algorithm has some sway above the
scientists because they rely so much on those indicators to further their careers,
to get a promotion, or get funding for their next research projects. So we have a
reciprocal relationship between the algorithm and the scientists, and this
creates a new construction of reality. So we can conclude that governance by
algorithms leads to behavioral adaptation in scientists, and one of these examples
that uses the Science Citation Index will be given from Franziska.
Franziska Sörgel: Thanks for the handover! Yes, let me start.
I’m focusing on reputation and authorship as you can see
on the slide, and first let me start with a quote
by Eugene Garfield, which says: “Is it reasonable to assume that if I cite a
paper that I would probably be interested in those papers which subsequently cite it
as well as my own paper. Indeed, I have observed on several occasions that people
preferred to cite the articles I had cited rather than cite me! It would seem to me
that this is the basis for the building up of the ‘logical network’ for the Citation
Index service.” So, actually, this Science Citation Index which is described here was
mainly developed in order to solve the problems of information retrieval. Eugene
Garfield, also founder of this Science Citation Index – short: SCI – noted or
began to note a huge interest in reciprocal publication behavior. He
recognized the increasing interest as a strategic instrument to exploit
intellectual property. And indeed, the interest in the SCI – and its data –
successively became more relevant within the disciplines, and its usage extended.
Later, [Derek J.] de Solla Price, another social scientist, asked or claimed for a
better research on the topic, as it currently also meant a crisis in science,
and stated: “If a paper was cited once, it would get cited again and
again, so the main problem was that the rich would get richer”, which is also
known as the “Matthew Effect”. Finally, the SCI and its use turned into a system
which was and still is used as a reciprocal citation system, and became a
central and global actor. Once a paper was cited, the probability it was cited again
was higher, and it would even extend its own influence on a certain topic within
the scientific field. So it was known that you would either read a certain article
and people would do research on a certain topic or subject. So this phenomenon would
rise to an instrument of disciplining science and created power structures.
Let me show you one example which is closely connected to this phenomenon
I just told you about – and I don’t know if here in this room there are any
astronomers or physicists? Yeah, there are few, okay.
That’s great, actually. So in the next slide, here,
we have a table with a time window from 2010 to 2016, and social
scientists from Berlin found out that the co-authorship within the field of physics
extended by 58 on a yearly basis in this time window. So this is actually already
very high, but they also found another very extreme case. They found one paper
which had roundabout 7,000 words and the mentioned authorship of 5,000. So, in
average, the contribution of each scientist or researcher of this paper who
was mentioned was 1.1 word. Sounds strange, yeah. And so of course you have
to see this in a certain context, and maybe we can talk about this later on,
because it has to do with Atlas particle detector, which requires high maintenance
and stuff. But still, the number of authorship, and you can see this
regardless which scientific field we are talking about, generally increased the
last years. It remains a problem especially for the reputation, obviously.
It remains a problem that there is such high pressure on nowadays researchers.
Still, of course, we have ethics and research requires standards of
responsibility. And for example there’s one, there’s other ones, but there’s one
here on the slide: the “Australian Code for the Responsible Conduct of Research”
which says: “The right to authorship is not tied to position or profession and
does not depend on whether the contribution was paid for or voluntary.
It is not enough to have provided materials or routine technical support,
or to have made the measurements on which the publication is based.
Substantial intellectual involvement is required.”
So yeah, this is, could be one rule to work with or to work by, to follow.
And still we have this problem of reputation which remains,
and where I hand over to Judith again.
Judith: Thank you. So we’re going to speak about strategic citation now. So if you
put this point of reputation like that, you may say: So the researcher does find
something in his research, his or her research, and addresses the publication
describing it to the community. And the community, the scientific community
rewards the researcher with reputation. And now the algorithm, which is like
perceived to be a new thing, is mediating the visibility of the researcher’s results
to the community, and is also mediating the rewards – the career opportunities or
the funding decisions etc. And what happens now and what is plausible to
happen is that the researcher addresses his or her research also to the algorithm
in terms of citing those who are evaluated by the algorithm, who he wants to support,
and also in terms of strategic keywording etc. And that’s the only thing which
happens new, might be a perspective on that. So the one thing new: the algorithm
is addressed as a recipient of scientific publications. And it is like far-fetched
to discriminate between so-called and ‘visible colleges’ and ‘citation cartels’.
What do I mean by that? So ‘invisible colleges’ is a term to say: “Okay, people
are citing each other. They do not work together in a co-working space, maybe, but
they do research on the same topic.” And that’s only plausible that they cite each
other. And if we look at citation networks and find people citing each other, that
does not have necessarily to be something bad. And we also have people who are
concerned that there might be like ‘citation cartels’. So researchers citing
each other not for purposes like the research topics are closely connected, but
to support each other in their career prospects. And people do try to
discriminate those invisible colleges from citation cartels ex post from looking at
metadata networks of publication and find that a problem. And we have a discourse on
that in the bibliometrics community. I will show you some short quotes how people
talk about those citation cartels. So e.g. Davis in 2012 said: “George Franck warned
us on the possibility of citation cartels – groups of editors and journals working
together for mutual benefit.” So we have heard about their journal impact factors,
so they... it’s believed that editors talk to each other: “Hey you cite my journal,
I cite your journal, and we both will boost our impact factors.”
So we have people trying to detect those cartels,
and Mongeon et al. wrote that: “We have little knowledge
about the phenomenon itself and about where to draw the line between
acceptable and unacceptable behavior.” So we are having like moral discussions,
about research ethics. And also we find discussions about the fairness of the
impact factors. So Yang et al. wrote: “Disingenuously manipulating impact factor
is the significant way to harm the fairness of the impact factor.” And that’s
a very interesting thing I think, because why should an indicator be fair? So the...
To believe that we have a fair measurement of scientific quality relevance and rigor
in one single like number, like their journal impact factor, is not a small
thing to say. And also we have a call for detection and punishment. So Davis also
wrote: “If disciplinary norms and decorum cannot keep this kind of behavior at bay,
the threat of being delisted from the JCR may be necessary.” So we find the moral
concerns on right and wrong. We find the evocation of the fairness of indicators
and we find the call for detection and punishment. When I first heard about that
phenomenon of citation cartels which is believed to exist, I had something in mind
which sounded... or it sounded like familiar to me. Because we have a similar
information retrieval discourse or a discourse about ranking and power in a
different area of society: in search engine optimization. So I found a quote by
Page et al., who developed the PageRank algorithm – Google’s ranking algorithm –
in 1999, which has changed since that a lot. But they wrote also a paper about the
social implications of the information retrieval by the PageRank as an indicator.
And wrote that: “These types of personalized PageRanks are virtually
immune to manipulation by commercial interests. ... For example fast updating
of documents is a very desirable feature, but it is abused by people who want to
manipulate the results of the search engine.” And that was important to me to
read because we also have like a narration of abuse, of manipulation, the perception
that that might be fair, so we have a fair indicator and people try to betray it.
And then we had in the early 2000s, I recall having a private website
with a public guest book and getting link spam from people
who wanted to boost their Google PageRanks,
and shortly afterwards Google decided to punish link spam in their
ranking algorithm. And then I got lots of emails of people saying: “Please delete my
post from your guestbook because Google’s going to punish me for that.” We may say
that this search engine optimization discussion is now somehow settled and it’s
accepted that Google's ranking is useful. They have a secret algorithm, but it works
and that is why it’s widely used. Although that journal impact factor seems to be
transparent it’s basically the same thing that it's accepted to be useful and thus
it's widely used. So the journal impact factor, the SCI and the like. We have
another analogy so that Google decides which SEO behavior is regarded acceptable
and punishes those who act against the rules and thus holds an enormous amount of
power, which has lots of implications and led to the spreading of content management
systems, for example, with search engine optimization plugins etc. We also have
this power concentration in the hands of Clarivate (formerly ThomsonReuters) who
host the database for the general impact factor. And they decide on who’s going to
be indexed in those journal citation records and how is the algorithm, in
detail, implemented in their databases. So we have this power concentration there
too, and I think if we think about this analogy we might come to interesting
thoughts but our time is running out so we are going to give a take-home message.
Tl;dr, we find that the scientific community reacts with codes of conduct to
a problem which is believed to exist. The strategic citation – we have database
providers which react with sanctions so people are delisted from the journal
citation records or journals are delisted from the journal citation records to
punish them for citation stacking. And we have researchers and publishers who adapt
their publication strategies in reaction to this perceived algorithmic power. But
if we want to understand this as a problem we don’t have to only react to the
algorithm but we have to address the power structures. Who holds these instruments in
in their hands? If we talk about bibliometrics as an instrument and we
should not only blame the algorithm – so #dontblamethealgorithm.
Thank you very much! applause
Herald: Thank you to Franziska, Teresa and Judith, or in the reverse order.
Thank you for shining a light on how science is actually seen
in its publications.
As I started off as well,
it’s more about scratching each other a little bit.
I have some questions here from the audience.
This is Microphone 2, please!
Mic2: Yes, thank you for this interesting talk. I have a question. You may be
familiar with the term ‘measurement dysfunction’, that if you provide a worker
with an incentive to do a good job based on some kind of metric then the worker
will start optimizing for the metric instead of trying to do a good job, and
this is kind of inevitable. So, don’t you see that maybe it could be treating the
symptoms if we just react about code of conduct, tweaking algorithms or addressing
power structures. But instead we need to remove the incentives that lead to this
measurement dysfunction. Judith: I would refer to this phenomenon
as “perverse learning” – learning for the grades you get but not for your intrinsic
motivation to learn something. We observe that in the science system. But if we only
adapt the algorithm, so take away the incentives, would be like you wouldn’t
want to evaluate research at all which you can probably want to do. But to whom would
you address this call or this demand, so “please do not have indicators” or… I give
the question back to you. laughs Herald: Okay, questions from the audience
out there on the Internet, please. Your mic is not working? Okay, then I go to
Microphone 1, please Sir. Mic1: Yeah, I want to have a provocative
thesis. I think the fundamental problem is not how these things are gamed but the
fundamental problem is that if we think the impact factor is a useful measurement
for the quality of science. Because I think it’s just not.
applause
Judith: Ahm.. I.. Mic 1: I guess that was obvious...
Judith: Yeah, I would not say
that the journal impact factor is a measurement of scientific quality
because no one has like a definition of scientific quality.
So what I can observe is only people believe this journal impact factor
to reflect some quality. Maybe they are chasing a ghost but I…
whether that’s a valid measure is not so important to me,
even if it were a relevant or a valid measure,
it would concern me how it affects science.
Herald: Okay, question from Microphone 3 there. Please.
Mic3: Thanks for the interesting talk. I have a question about
the 5,000 authors paper. Was that same paper published
five thousand times or was it one paper with ten page title page?
Franziska: No, it was one paper ...
... counting more than 7,000 words. And the authorship,
so authors and co-authors, were more than 5,000.
Mic3: Isn’t it obvious that this is a fake?
Franziska: Well that’s what I meant earlier
when saying, you have to see this within its context. So physicists are working
with this with Atlas, this detective system. As there were some physicists in
the audience they probably do know how this works. I do not. But as they claim
it’s so much work to work with this, and it, as I said, requires so high
maintenance it’s... They obviously have yeah...
Mic3: So everybody who contributed was listed?
Judith: Exactly, that’s it. And if this is ethically correct or not, well, this is
something which needs to be discussed, right? This is why we have this talk, as
we want to make this transparent, and contribute it to an open discussion.
Herald: Okay, I’m sorry guys. I have to cut off here because our emission out
there in space is coming to an end. I suggest that you guys
find each other somewhere, maybe in the tea house or...
Judith: Sure. We are around, we are here.
Herald: You are around. I would love to have lots of applause for these ladies,
for it really lights on how these algorithms
not or are working. Thank you very much!
Judith: Thank you!
postroll music
subtitles created by c3subtitles.de in the year 2018