Open access citation effect illusory

Journal articles made freely available online are accessed more than articles with a subscription cost but are not cited more. This is the controversial early finding of what will be a four-year study at Cornell University. The open access (OA) lobby has slammed the publication of the preliminary report as premature.

Past studies have shown that OA literature is cited more than non-OA literature but it has not been clear whether this is due to its free availability or factors such as more popular papers or authors being made OA.

For the Cornell study, 247 articles published in 11 scientific journals were randomly assigned OA or subscription access, and downloads, visitors and citations counted. The OA articles had 89% more full-text downloads, 42% more PDF downloads and 23% more unique visitors, although 24% fewer abstract downloads in the first six months after publication.

Yet OA articles were no more likely to be cited in the first year after publication, with 59% of OA articles cited nine to 12 months after publication compared with 63% of subscription access articles.

Citation advantage
In their report, published in the British Medical Journal in August, the researchers concluded: “The citation advantage from open access reported widely in the literature may be an artefact of other causes.”

Fiona Godlee, editor of the BMJ, said: “This is, as far as I know, the first randomised trial of open access. Just as with medical interventions, we can’t properly evaluate effects without random allocation, so this is an advance on the observational studies previously reported.

“The fact these initial results suggest OA increases usage but not citations fits with the way in which citations are largely generated by people who already have access to the literature and for whom OA is therefore less of a benefit.”

However, she cautioned that the results were from one trial, in one discip line, and were preliminary results after only one year of research.

John Sack, publisher and director of HighWire Press, welcomed the paper. “This paper is interesting for several reasons,” he said. “First, it is by a researcher working in the field rather than an advocate of one position or another on the topic of open access. The researcher also, because of his previous career as a librarian and reader of journals, understands the debate and the ‘use cases’.

“Second, it is a prospective, randomised study. It shows one type of effect (usage) but not the other type of effect (citations). This encourages us to dig into the data and learn more about who gets value and when.

“Perhaps most importantly, the study is continuing and so we will be able to look at data for different fields and for a different time-horizon. If the effects are different for different fields, again we learn something about who gets value and when.”

Some observers believe it is far too early to draw any conclusions from the research. Gunther Eysenbach, author of the influential paper Citation Advantage of Open Access Articles, said the follow-up time was much too short.

“Some other variables which should be predictors for higher citations – eg articles with a press release – were also negative,” he pointed out. “Before one can declare a negative result, one would have to wait at least three years for citations to accumulate and then test for non-inferiority.”

The study’s co-author Phil Davis said the preliminary results were published because of the importance of the issue and the stark contrast between these results and those in prior studies.

“For example, Eysenbach detected more than a two-fold difference in the incidence of OA articles being cited in the first four to 10 months after publication,” he said. “Based on the effect size reported in his and other studies, we should have seen a significant OA effect by the end of the first ye ar.”

Matthew Cockerill, publisher at BioMed Central, said that the citations an article received in its first year after publication were generally a very small fraction of those it ultimately received. “The fact that no effect is seen at this point is not a great surprise,” he said, “and certainly doesn’t provide a strong basis for making general statements about the overall effect of access on citations.”

He added that the Web of Science, used to identify citations of the articles in the study, was “unfortunately anything but comprehensive in its coverage, and in particular its coverage of recently launched journals is patchy. It would therefore be advisable to estimate citation counts using other sources too, such as Scopus and Google Scholar.”

Convincing
Michael Kurtz, a scientist who has worked on the role of the ArXiv e-print archive and its effect on citations, was more enthusiastic about the research. “I find the Davis article quite convincing,” he said. “Of course, the study will be better after several years, but if the OA effect were strong and causal, one ought see the signal in his data by now.”

Kurtz added that the controversy was not about whether articles posted online were more heavily cited than those not posted – they clearly are – but whether the increase in citations was caused by the posting. “There has been a steady stream of papers trying to show this causality,” he said. “None has done so. Davis’s study is a fully randomised trial and shows no causal effect on citations of an article’s being OA. This should end the controversy; for me, it does.”

Jonathan Wren, author of a paper published in the BMJ on OA to medical information, believes citation research is of limited value and the real issues with OA lay elsewhere. “The higher priority question is understanding whether the financial barriers that OA publishing may create for authors is affecting research and how,” he said. “Prices have skyrocketed to around $2,000 to $3,000 per paper and apparently are not done.”

Since submitting their paper to the BMJ (with citation data from 2 January, 2008), the authors have run update analyses. According to Davis, as of 3 August 2008 (15 to 18 months after article publication) the effect of randomised OA on citations remains insignificant.

“OA and subscription-access articles both average 3.8 citations,” he said. “Even after an additional six months, we still find no OA effect on citations. Nonetheless, we plan to gather more citation data and to re-examine the issue after allowing even more time to pass.”

The report is at tinyurl.com/5au8ywo

Like this story? Spread the news by clicking below:

 del.icio.us     Digg this     reddit!