Blog
About

In:  Impact Factor  

How can academia kick its addiction to the impact factor?

The impact factor is academia’s worst nightmare. So much has been written about its flaws, both in calculation and application, that there is little point in reiterating the same tired points here (see here by Stephen Curry for a good starting point).

Recently, I was engaged in a conversation on Twitter (story of my life..), with the nice folks over at the Scholarly Kitchen and a few researchers. There was a lot of finger pointing, with the blame for impact factor abuse being aimed at researchers, at publishers, funders, Thomson Reuters, and basically any player in the whole scholarly communication environment.

As with most Twitter conversations, very little was achieved in the moderately heated back and forth about all this. What became clear though, or at least more so, is that despite what has been written about the detrimental effects of the impact factor in academia, they are still widely used: by publishers for advertising, by funders for assessment, by researchers for choosing where to submit their work. The list is endless. As such, there are no innocents in the impact factor game: all are culpable, and all need to take responsibility for its frustrating immortality.

The problem is cyclical if you think about it: publishers use the impact factor to appeal to researchers, researchers use the impact factor to justify their publishing decisions, and funders sit at the top of the triangle facilitating the whole thing. One ‘chef’ of the Kitchen piped in by saying that publishers recognise the problems, but still have to use it because it’s what researchers want. This sort of passive facilitation of a broken system helps no one, and is a simple way of failing to take partial responsibility for fundamental mis-use with a problematic metric, while acknowledging that it is a problem. The same is similar for academics.

Oh, I didn’t realise it was that simple. Problem solved.

Eventually, we agreed on the point that finding a universal solution to impact factor mis-use is difficult. If it were so easy, there’d be start-ups stepping in to capitalise on it!

(Note: these are just smaller snippets from a larger conversation)

What some of us did seem to agree on, in the end, or at least a point remains important, is that everyone in the scholarly communication ecosystem needs to take responsibility for, and action against, mis-use of the impact factor. Pointing fingers and dealing out blame solves nothing, and just alleviates accountability without changing anything, and worse, facilitating what is known to be a broken system.

So here are eight ways to kick that nasty habit! The impact factor is often referred to as an addiction for researchers, or a drug, so let’s play with that metaphor.

  1. Detox on the Leiden Manifesto

The Leiden Manifesto provides a great set of principles for more rigorous research evaluation. If these best-practice principles could be converted into high level policy for institutes and funders, with a major push for their implementation coming from the research community, we could see a real and great change in the assessment ecosystem. With this, we will see a concomitant change in how research develops and interacts with society. Evaluation criteria must be based on high quality and objective quantitative and qualitative data, and the Leiden Manifesto lays out how to do this.

  1. Take a DORA nicotine patch

The San Franciso Declaration on Research Assessment (DORA) was started in 2012 by a group of Editors and publishers of scholarly journals in order to tackle malpractice in research evaluation. It recognised the inadequacies of the impact factor of scientific quality, and provided a series of recommendations for improving research evaluation. These include: eliminating the use of journal-based metrics; assessing research based on its own merits;  and exploring new indicators of significance.

To date, 7985 individuals and 589 organisations have signed DORA. That is less than the number of researchers boycotting Elsevier, and the number of global open access policies, respectively, so there is still much scope for communicating and implementing these recommendations.

  1. Attend ‘objective evaluation’ clinics and bathe in a sea of metrics

In 2015, a report called ‘The Metric Tide’ was published following an Independent Review of the Role of Metrics in Research Assessment and Management. This was set up in April 2014 to investigate the current and potential future roles that quantitative indicators can play in the assessment and management of research.

They found that peer review and qualitative indicators should form the basis for evaluating research outputs and individuals, with careful use of metrics as a supplement. This will help to capture more diverse aspects of research, and limit concerns arising from the gaming and mis-use of metrics such as the impact factor. They also advocated the responsible use of metrics based on dimensions of transparency, diversity, robustness, reflectivity, and humility – 5 traits, neither of which the impact factor possesses.

struggling-scientist

  1. Vape your way to a deeper understanding of impact factors

To summarise well-documented limitations of the impact factor (from DORA):

  1. Citation distributions within journals are highly skewed;
  2. The properties of the Journal Impact Factor are field-specific: it is a composite of multiple, highly diverse article types, including primary research papers and reviews;
  3. Journal Impact Factors can be manipulated (or “gamed”) by editorial policy;
  4. Data used to calculate the Journal Impact Factors are neither transparent nor openly available to the public.

Recent research has also shown that impact factors are strongly auto-correlated, becoming a sort of self-fulfilling prophecy. It is deeply ironic that researchers, supposedly the torch-bearers of reason, evidence, and objectivity, persistently commit to using a metric that has been so consistently shown to be unreasonable, secretive, and statistically weak. To learn more, see Google.

Understanding thine enemy is the first step to being able to defeat them.

  1. Chew on the gummy content of the paper

Knowledge is nicotine for researchers. There will never ever be a metric that surpasses the value of assessing the quality of a paper than reading it. ‘Read the damn paper’ has even become a bit of a rallying cry for the anti-impact factor community, which makes perfect sense. However, there are often situations on which assessment of huge swathes of papers and other research outputs has to be conducted, and therefore short-hand alternatives to reading papers are used as proxies to measure the quality of papers – such as the impact factor, or the journal title. This becomes a problem when quality and prestige diverge, as is very common, as they are no longer reflective of the same traits. Solutions exist, such as employing greater numbers of people in assessments, submission only of key research outputs, which enable the process of being able to digest the content of an article or other output and being able to make more-informed assessments of research. It is vastly unfair and inappropriate that researchers, funders, and other bodies are put in a position where they are unable to commit to these and forced to use inappropriate shortcuts instead. However, when time and volume is not an issue, there is simply no excuse for evaluating work based on poor proxies.

Credit: Hilda Bastian
Credit: Hilda Bastian
  1. Just quit. Go cold turkey.

As someone who used to smoke, I finally quit by going cold turkey. Partially because I could no longer afford to keep up the habit as a student, but that’s besides the point. The point is to make a personal commitment to yourself that you will no longer succumb to the lures of the impact factor. Reward yourself with cupcakes and brownies. You owe it to yourself to be objective, to be critical, and to be evidence informed about your research, and this includes how you evaluate your colleagues’ work too. Commitments like this can be contagious, and it always helps to have the support of your colleagues and research partners. Create a poster like “This is an impact factor free work environment” and stick it somewhere everyone can see!

Something like this, but with more or less raptors depending on preference.
Something like this, but with more or less raptors depending on preference.
  1. Don’t hang around other impact factor junkies

The first rule of impact factors is we don’t talk about impact factors (irony of this post fully appreciated). This is ‘how to kick an addiction 101’. When you quit smoking/drugs/coffee, the last thing you want is to be hanging around others who keep doing it. It’s bad for your health, and just drags you right back down the path of temptation. If someone insists on using the impact factor around you, explain to them everything in this post. Or just leave. They’re simply adopting bad practices, and you don’t want to or have to be part of that. If it’s your superior, a long frank discussion about about the numerous problems and alternatives of the impact factor is well worth your time. Scientists are well known for being completely reasonable and open to these sorts of discussion, so no problems there.

  1. Take a methadone hit of sweet sweet altmetrics

While this has sort of been covered by the first three points, but altmetrics, or alternative metrics are a great way of assessing how your research has been disseminated on social channels. As such, they are a sort of pathway or guide to ‘societal impact’, and provide a nice compliment to citation counts, which are often used as a proxy for ‘academic impact’. Importantly, they are at the article level, so do not suffer the enormous shortcomings of journal-level metrics such as the impact factor, and offer a much more accurate insight into how research is being re-used.

What other solutions can we implement in eliminating the impact factor, and making academic assessment and publishing a more fair, transparent, and evidence-informed process? The Metrics Tide report, DORA, and the Leiden Manifesto are all great steps towards this goal, but the question still remains of how we embed their recommendations and principles in academic culture.

We should be very aware that there is absolutely nothing to lose from employing these recommendations and partial solutions. What we can gain though is an enriched and informed process of evaluation, which is fair and benefits everyone. That’s important.

9 thoughts on “How can academia kick its addiction to the impact factor?”

  1. Excellent post Bjorn! The problem is just as much about the *perception* of how research assessment is done, as you say. We need to move to a much broader means of assessing research and researchers instead of relying almost entirely on *where* they publish.misuse of Impact Factors is not the cause of the problem, it’s an effect of it.

  2. I think it is important not to conflate JIF and journal prestige. For example some people may want to publish in Nature because it has an IF of ##?, but many others probably want to publish in it because it is Nature. Once one gets into the large mass of journals that both have little broad reputation or very high IF then, perversely researchers may well use the IF more at the moment to discriminate between them and decide where to send a manuscript. Which is both sloppy and probably has little impact on the ‘quality’ of their own CV.
    But my point is there has always been a certain prestige hierarchy between journals, and JIF has merely formalised this in a numeric sense (we can argue about how well it does this and structural flaws, but it doesn’t invalidate the underlying point). Merely removing JIF would not address this. I can think of the discipline specific journals in my field (and I’m sure everyone can) which researchers are pleased to get papers published in, not necessarily (just) for the JIF, but for the general prestige of having a paper in a journal recognised by ones peers as being of generally good quality.
    Therefore if JIF was removed overnight it may see some change in publishing/submitting practice, but people in my field would still want to publish in Geology, Geophysical Research Letters, Water Resources Research etc, because they are seen as quality journals independent of whatever number they have for their JIF.
    So what I’m saying is that removal of JIF would get rid of a lot of nonsense and metrics, but it would be highly unlikely to change the fact that some journals have a better reputation than others and many people will want to publish their papers in journals with a good reputation. Therefore publishing practice may not see a dramatic shift in behaviour in the absence of JIF.

  3. Hi Jon,

    I am more and more convinced that this talk against impact factors could in fact be dangerous. In a similar way that requesting open access has backfired and led to the gold OA model that has been a terrible development because it has pacified the community’s momentum without addressing the core problem in academia. This core problem is simply that Universities waste billions for subscriptions (or APCs) instead of developing and offering for free a modern scholarly communication infrastructure that will include —among other essential services now partially provided by startups that eventually become absorbed by big publishers— a sophisticated and efficient reward system.

    Publishers are a thing of the past. The sooner we realise that and start organising our communities and associations around public infrastructures the better. One big problem is that publishers throw every now and then some crumbs to academics, politicians, associations, trying to keep everyone happy and calm. But hopefully, infrastructure people will eventually wake up and Universities will start offering, apart from an email account, free services for co-authoring, reference managers and other discovery tools, data-mining algorithms, software code, data visualisation tools, cloud archive for code, data and manuscripts, peer review, usage statistics, etc, etc. Until all this infrastructure is provided for free by institutions to their faculty we will still need publishers, impact factors, and promising startups by former academics who decided to abandon research to save the world. I think that asking researchers, especially young ones, to stop talking about impact factors (as suggested in most of your points) and concentrate on how to find the money for APCs is asking them to put their careers at risk and is utterly unethical.

    1. What kind of researchers are we “educating” if we tell young, curious, aspiring scientists that they have to care about the impact factor of where they publish their research results? Guido Guidotti, Harvard biochemistry professor told me of a post-doc applying to join his lab. When he asked what project the candidate was interested in, the young man said “I don’t care, as long as I get a paper in Nature or Cell in the first year.” Are those the kinds of researchers who are going to genuinely move science forward? I have spoken to so many young researchers who have told me that they are completely disillusioned by science because they thought it was about finding solutions to problems and are taught along the way not to share so that they can be the first to publish and maximize their brownie points with a high impact factor journal. We are losing many bright scholars and promoting those who are best at playing the reputation game. But there are also young researchers out there fighting for a more open and rigorous evaluation system and still making a career in science. Maybe their path is harder, but I don’t think that it is “unethical” to point to them as role models.

      1. Hi Stephanie,
        there is absolutely no need to educate us! I didn’t receive any education to use my institutional email account. It was just provided for free to me. I use overleaf to co-author my papers, but if my university offered me a free alternative I would immediately switch to it. I publish my papers in high IF journals, but if my institutional repository/platform offered an alternative reward system recognised by evaluating committees I would immediately forget about journals and impact factors. We don’t need education, we need free, public, federated alternatives. We are not responsible for the system and I insist that it is not appropriate to ask researchers to save the world by switch from “bad” publishers, to “good” publishers. To us they are all these same. Commercial enterprises that take advantage of a gap of services that were meant to be offered for free by research institutes. There is more than enough public money for that. That’s why I say that I don’t want my University to sign DORA, I just want them to stop paying subscriptions and APCs and to offer me the tools I need to do my research (and receive proper recognition for it). In the meantime, I will continue to prefer a Nature publication than any open access journal made by —former— researchers for researchers! And I seriously question the motivation of anyone who is trying to blame me for this choice 🙂

    2. Hi Jon,
      of course it doesn’t do any harm to support such initiatives. I just doubt it does any good. I direct you to a recent exchange of comments I had with David Crotty from the Scholarly Kitchen: https://scholarlykitchen.sspnet.org/2016/03/23/ask-the-chefs-what-is-the-biggest-misconception-people-have-about-scholarly-publishing/

      There, he correctly notes that whatever alternatives to the IF we come up with, publishers will simply adapt. We ask for OA, publishers are there, we ask for altmetrics, publishers provide, we ask for open peer review, no problem. We call them “bad”, “good” publishers immediately appear to fill in the gap and get their market share. It’s just useless. You can’t compete with that. And I seriously doubt who really wants to compete and bring change and who is just trying to invent ways to shift authors’ behaviour in order to get a piece of the pie.

      I can only repeat that the solution is to forget about publishers altogether and offer the necessary infrastructure for research preparation, discovery, analysis, writing, publication, outreach, assessment and archiving for free as standard services offered by Universities and research institutions.

  4. “We should not do so much fuss on Journal Impact Factor.If you have quality (true/sathya) paper every one will accept it.” Even policy maker go out of the way and help you.

  5. Thanks for this important post, but unless academic administrators, regulatory bodies, and funding agencies keep asking for this, I do not foresee any solution. However, as suggested by you, it will be good to adopt alternatives to see the real impact of research.

Comments are closed.