ISI Impact Factor Data Under Fire (again)

An editorial appearing in the Journal of Cell Biology by Mike Rossner, Heather Van Epps, and Emma Hill entitled Show me the data reports the inability for the authors to verify published impact factors using data provided by ISI. While it is common to read about the quirks of the impact factor, the authors question the underlying validity of the data used to calculate those impact factors and therefore the validity of the metrics that are published using it.

The authors, from The Rockefeller University Press, The Journal of Experimental Medicine, The Journal of Cell Biology, highlight their unsuccessful efforts to replicate ISIs published impact factors for these journals, which they serve as directors/editors. They reveal numerous and serious errors in several data sets provided by ISI.

When we requested the database used to calculate the published impact factors (i.e., including the erroneous records), Thomson Scientific sent us a second database. But these data still did not match the published impact factor data. This database appeared to have been assembled in an ad hoc manner to create a facsimile of the published data that might appease us. It did not.

When we examined the data in the Thomson Scientific database, two things quickly became evident: first, there were numerous incorrect article-type designations. Many articles that we consider "front matter" were included in the denominator. This was true for all the journals we examined. Second, the numbers did not add up. The total number of citations for each journal was substantially fewer than the number published on the Thomson Scientific, Journal Citation Reports (JCR) website (http://portal.isiknowledge.com/, subscription required). The difference in citation numbers was as high as 19% for a given journal, and the impact factor rankings of several journals were affected when the calculation was done using the purchased data (data not shown due to restrictions of the license agreement with Thomson Scientific).

It became clear that Thomson Scientific could not or (for some as yet unexplained reason) would not sell us the data used to calculate their published impact factor. If an author is unable to produce original data to verify a figure in one of our papers, we revoke the acceptance of the paper. We hope this account will convince some scientists and funding organizations to revoke their acceptance of impact factors as an accurate representation of the quality—or impact—of a paper published in a given journal.

If the problems that these editors encountered in their research are indeed accurate and widespread, the qualitative and evaluative decisions that rely in part on ISI's published impact factors (library purchasing; promotion and tenure; hiring; where to submit manuscripts) could now be considered suspect. Sphere: Related Content