Impact Factor? Shmimpact Factor! The Journal Impact Factor, Modern Day Literature Searching, and the Publication Process
by Leslie Citrome, MD, MPH
Dr. Citrome is Professor of Psychiatry, New York University School of Medicine, and Director, Clinical Research and Evaluation Facility, Nathan S. Kline Institute for Psychiatric Research, Orangeburg, New York
Financial Disclosures: Dr. Citrome is a consultant for, has received honoraria from, or has conducted clinical research supported by the following: Abbott Laboratories, AstraZeneca Pharmaceuticals, Barr Laboratories, Bristol-Myers Squibb, Eli Lilly and Company, GlaxoSmithKline, Janssen Pharmaceuticals, Jazz Pharmaceuticals, and Pfizer Inc.
The journal impact factor is a measure of the citability of articles published in that journal—the more citations generated, the more important that article is considered to be, and as a consequence the prestige of the journal is enhanced. The impact factor is not without controversy, and it can be manipulated. It no longer dominates the choices of journals to search for information. Online search engines, such as PubMed, can locate articles of interest in seconds across journals regardless of high or low impact factors. Editors desiring to increase their influence will need to focus on a fast and friendly submission and review process, early online and speedy print publication, and encourage the rapid turnaround of high-quality peer reviews. Authors desiring to have their results known to the world have never had it so good—the internet permits anyone with computer access to find the author’s work.
Key Words: journal impact factor, peer review, publication, PubMed, searching
Counting citations to a journal article sounds like an intuitive way to gauge the importance of a research report, and it was Eugene Garfield who proposed the concept in a landmark 1955 Science article. Five years later, Garfield founded the Institute for Scientific Information (ISI).[2,3] ISI maintains bibliographic databases, produces citation indexing, and conducts analyses on these. ISI publishes an annual Journal Citation Report, which provides an impact factor for each journal in its database. The higher the impact factor, the more influential that journal is thought to be. Impact factor is calculated as the mean citation rate of all the articles in the journal. The assumption is that journal articles that are often cited by other journal articles are more important than journal articles that are not cited as often. The journal that produces the most citable journal articles wins. This is potentially very important for potential advertisers who want to be associated with prestigious journals, and for academic departments at medical schools that are evaluating faculty for promotion and tenure. Journals will use their impact factors in their own solicitations for additional papers and for advertisers. Some include it on the cover of every issue (for example, Schizophrenia Research). Libraries may look at journal impact factor when deciding which subscriptions to renew. ISI was acquired by Thomson Scientific & Healthcare in 1992, and the company is now known as Thomson Scientific, a part of Thomson Corporation, one of the largest for-profit information companies in the world, and the producer of the annual drug compendium provided free to US physicians, the Physicians’ Desk Reference (see also http://scientific.thomson.com). The journal impact factor is big business.
Garfield himself has produced over 1500 articles, many available at www.garfield.library.upenn.edu. His contributions to information science are paradoxically incalculable. The journal impact factor, however, remains controversial, and its relevance in today’s internet era is less certain. We will review some of the issues surrounding the use of the journal impact factor, followed by a review on how researchers and clinicians conduct literature reviews. The role of medical books as sources of information is discussed, followed by suggestions to medical journal editors and authors.
Commentary on the use of the impact factor
A retrospective analysis of trends for seven medical journals and their impact factors was undertaken and revealed a sharp rise in scores over 1994 to 2005. Some increases were as much as 500 percent (for the Canadian Medical Association Journal). The article noted that because the calculation of the impact factor consists of dividing “number of citations generated” by “number of citable articles,” some journals can increase their impact factor by publishing fewer citable articles. This appears to have occurred in some cases. Journal editors were also interviewed and described the active recruitment of high impact articles. Boosting the journal’s media profile was also identified as a strategy to increase the impact factor by increasing the visibility and citability of its contents.
However, a journal impact factor does not guarantee that an individual article it contains is of high impact. There is wide variability in citation rates for individual articles; articles in the most cited half of articles in a journal are cited 10 times as often as the least cited half. Moreover, journal impact factors depend on the type of scientific work reported; high impact factors are more often observed in journals covering large areas of basic research that use many references per article.6 For the individual author, publishing in a high-impact journal may be prestigious in the eyes of promotion committees, but there is no guarantee that the work would be cited more often than if it had appeared in a lower-impact journal. This latter view is not shared by all, with a study finding that among 204 published articles whose data was originally presented at the 1991 meeting of the Society for Academic Emergency Medicine, the strongest predictor of citations per years was the impact factor of the original publishing journal, followed by the newsworthiness score (90% as strongly), and a subjective quality score (62% as strongly).
Conducting literature reviews
Newly minted physicians may not be aware of the significant obstacles and delays inherent in the literature searches of yesteryear. Not so long ago, the only accessible way of systematically searching for relevant journal articles was by going to the medical library and pulling down from the reference shelves heavy tomes of Index Medicus, a multivolume printed index of what has been published in the major medical journals. With luck, the reader would stumble upon a recent review of the topic of interest and this would generate other lists of articles to look up. Books were another source of bibliographic information, but often lagging several years behind the journal literature. Receiving a Xeroxed copy of the table of contents for the principal general and specialty journals was one of the principal means of keeping up. Because of time constraints and logistical reasons, the number of journals that could be routinely picked over was limited.
Today anyone with an internet connection can access http://pubmed.gov or http://highwire.org, two medical literature search engines that offer free access to the National Library of Medicine Medline, a very large medical literature database. PubMed is the most widely used portal and has been available to the public since 1995, with usage of 70,000,000 searches per month. In addition to the free access to abstracts, many journal articles are available for free download. Clinicians with hospital or university affiliations can get broader access to even more journal titles. Physicians in New York State have free access, upon registration, to the New York state Library and its full text electronic journal holdings (see http://www.nysl.nysed.gov). Low impact and high impact journals are both indexed, and the clinician will encounter search “hits” in either category, and may not care which category it is, if the answers they are searching for are easily found. Moreover, in a comparison of review articles published in peer-reviewed scholarly journals with those found in “throwaway” journals, peer-reviewed article titles were judged less relevant to clinical practice than throwaway journal article titles.
Of potentially large impact in terms of influencing clinicians is the selection of papers featured on clinically oriented portals such as http://www.medscape.com. Although researchers and academics will routinely go through lists generated by PubMed, the clinician seeking a quick answer may not. Hence, a review posted on Medscape may be read by far more people than any single article printed in a relatively obscure (to the clinician) but nevertheless high impact factor specialty journal. The editors of PloS Medicine have suggested that a “download impact factor” may be useful.
A relatively new search portal is http://scholar.google.com. When searching for scholarly publications, this search engine will provide the number of citations made to that article, providing competition to the services provided by ISI. It will also provide information about the articles that cite the article of interest.
Are books obsolete?
Books and monographs continue to be written, but they are not indexed the same way as journal articles and are relatively inaccessible. One exception is the emergence of books that may appear on searches of http://scholar.google.com, but the lists produced are incomplete. In general, potential readers would not know of the book unless they receive an advertisement for it or if it is cited in a journal article. It is possible to search the site of the bookseller http://www.amazon.com for subjects and authors, but the results are usually limited to recent publications, and the lists produced can contain many extraneous results.
For readers who want the most up-to-date information, books are not ideal. The information is substantially older than journal articles because of the length of time it takes to prepare and edit the manuscript and print and distribute the actual book. Handbooks and guides remain important to publish, but what of the books whose contents are essentially collections of review articles? Unless they get indexed online, their readership will remain small, and their impact minimal.
Advice to editors
The quest for a higher impact factor seems dominant on the minds of many journal editors. Solicitation of potentially high impact original research findings may not be possible for the less prestigious journals; however, offering a friendly submission process and rapid review will encourage potential authors. Rapid review can be encouraged among peer reviewers by offering incentives, such as CME credit (3 credits was a pleasant surprise from the Archives of Internal Medicine and from the Southern Medical Journal) or access for a limited time to online versions of journals. Regular reviewers should be rewarded with a complementary subscription. For peer reviews that are especially urgent, payment for such a service could serve as an effective inducement.
Offering online early publication and rapid print publication is also highly desirable in attracting submissions. Increasing a journal’s exposure may mean offering free internet access to that journal for everyone, even if only for a limited period of time. Partnering with a medical society, either general or specialty, may also increase circulation. Co-publication of selected articles with an online portal, such as Medscape, will increase readership several-fold.
Although it may sound obvious, providing the author with a complimentary print copy of the journal issue in which their article appears, as well as a PDF copy for their files, is important for goodwill. This does not always happen, and I have my own anecdote to tell of a small specialty journal published by a large publishing company refusing to provide me with any copy whatsoever of my own contribution. The editor of that journal, who has since left that post, informed me that he and his editorial board no longer receive a complimentary subscription to their own journal.
Some journals conduct the peer review process in a double-blind fashion, although the identity of the author is often easy enough to determine based on the topic and supporting references cited in the paper. Other journals keep only the identity of the peer reviewer confidential, but in many circumstances it is possible for the experienced author to accurately determine who the reviewer is, based on the reviewer’s arguments and the additional suggestions and citations provided. A different approach has been taken by http://www.biomedcentral.com, an open-access online publisher of scholarly articles. The pre-publication submission, peer reviews, and revisions are published along with the final manuscript. Everything is public. This largely does away with slipshod peer reviews and the inappropriate comments that find their way into anonymized reviews. The open process also provides for the reviewers to be recognized for their otherwise thankless task of writing a careful critique of a manuscript.
Advice to authors
The journal impact factor is one consideration when selecting a potential journal for a manuscript. However, not all papers are suitable for Nature, Science, The New England Journal of Medicine, or JAMA. For these papers, the issue of which journal to submit to comes down to whether they are indexed, have a fast and friendly review process, and are reasonably rapid in the process of online posting and print publication. To increase visibility, and chances of automated “web-crawlers” finding the manuscript, abstracts should contain all key words that may be associated with the topic discussed. No matter how important or novel the findings are, the article will not be cited (let alone read) if no one notices. A catchy title also helps.
For the past several decades, the journal impact factor has been a marker of a medical journal’s prestige. It is a measure of the citability of articles published in that journal—the more citations generated, the more important that article is considered to be, and, as a consequence, the prestige of the journal is enhanced. Journal impact factors influence what manuscripts are submitted, what libraries are willing to spend on subscriptions, and what advertisers may be willing to pay. A review of the impact factor literature provides an outline of the main controversies surrounding the use of the journal impact factor: It can be manipulated, it does not guarantee quality of a journal’s content, and a high impact factor may or may not increase the author’s citability when publishing in that journal.
The conduct of literature searches has been revolutionized with the availability of the internet. Online search engines, such as PubMed, are in common use by academics and clinicians alike. Portals, such as Medscape, are used by clinicians to keep abreast of the fast-moving medical literature. Book chapters, not yet searchable in the same way as the current medical literature, may not have the impact they once had.
Editors desiring to attract high quality articles and increase their influence on academia and the clinic will need to provide a fast and friendly submission and review process, as well as early online and speedy print publication, and encourage the rapid turnaround of high-quality peer reviews. Authors desiring to have their results known to the world have never had it so good—the internet permits anyone with computer access to find the author’s work.
1. Garfield E. Citation indexes for science. Science 1955;122:108–11.
2. Garfield E. Journal impact factor: A brief review. CMAJ 1999;161:979–80.
3. Garfield E. The history and meaning of the journal impact factor. JAMA 2006;295:90–3.
4. Porta M, Fernandez E, Bolumar F. Commentary: The ‘bibliographic impact factor’ and the still uncharted sociology of epidemiology. Int J Epidemiol 2006;35:1130–5.
5. Chew M, Villanueva EV, Van Der Weyden MB. Life and times of the impact factor: Retrospective analysis of trends for seven medical journals (1994–2005) and their Editors’ views. J R Soc Med 2007;100:142–50.
6. Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314:498–502.
7. Seglen PO. Causal relationship between article citedness and journal impact. J Am Soc Information Sci 1994;45:1–11.
8. Callaham M, Wears RL, Weber E. Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer reviewed journals. JAMA 2002;287:2847–50.
9. Vanhecke TE, Barnes MA, Zimmerman J, et al. PubMed vs. HighWire Press: A head-to-head comparison of two medical literature search engines. Comput Biol Med 2006 Dec 19; [Epub ahead of print] doi:10.1016/j.compbiomed.2006.11.012
10. Rochon PA, Bero LA, Bay AM, et al. Comparison of review articles published in peer-reviewed and throwaway journals. JAMA 2002;287:2853–6.
11. PloS Medicine Editors. The impact factor game. It is time to find a better way to assess the scientific literature. PloS Medicine 2006;3:e291. doi:10.1371/journal.pmed.0030291
12. Godlee F. Making reviewers visible: Openness, accountability, and credit. JAMA 2002;287:2762–5.