Altmetrics offers all kinds of insights into how a piece of research has been communicated and cited. In 2014 Altmetric.com added policy document tracking to its sources of attention, offering another valuable insight into how research outputs are used post-publication. At the University of Sheffield, we thought it would be useful to explore the Altmetric.com data for policy document citations to see what impact our work is having on national and international policy.
We analysed all published research from authors at the University of Sheffield indexed in the Altmetric.com database:
- There were 96,550 research outputs
- We identified 1463 pieces of published research cited between one and 13 times in policy
- This represented 0.65% of our research outputs
- Of these 1463 artefacts, 21 were cited in five or more policy documents
- The vast majority – 1185 documents – were cited just once.
From our sample we found 92 research articles cited in three or more policy documents. Of those 92, medicine, dentistry and health had the greatest policy impact, followed by social science and pure science.
The time between publication and citation
We also wanted to explore whether research published by the University of Sheffield had a limited time span between publication and policy citation. We looked at the time lag and found it ranged from just three months to 31 years. This highlighted a long tail of publications influencing policy, something we would have struggled to identify prior to Altmetric.com without manual trawling.
The earliest piece of research from our sample to be cited in policy was published in 1979 and took until 2010 before receiving its first policy citation. We manually checked the records as we found many pre-1979 publications to have been published much later, often this century. This is likely due to misreported data in the institutional dataset, giving a false date; highlighting the need to manually check such records for authenticity. The shortest time between research publication and policy citation was a mere three months: a paper published in November 2016 and first cited in National Institute for Health and Care Excellence (NICE) policy in January 2017.
Errors in the data
The Altmetric.com reports are only as good as the data they analyse, and our research did uncover some errors. Looking at those 21 papers with more than five policy document citations, we found seven were not fit for inclusion. One such example was identified when we discovered research papers had been attributed to the University of Sheffield when the authors were not, in fact, affiliated to the university. As this data is sourced from our research publications system, we assume this was a mistake made by the author (this can happen when authors incorrectly accept as their own papers suggested to them by the system). While this was almost certainly a genuine error, and may have been rectified later, the system had not yet updated to take account of such corrections.
Another of these papers was mistakenly attributed to an author who had no direct involvement in the paper but who was part of a related wider research project. Another of the publications was excluded due to it not, in fact, having actually been cited in the relevant policy document. One of the papers that was included belonged to an author not at Sheffield at the time of publication, but who has since joined the institution. This showed that Altmetric.com’s regular updates were able to discover updated institutional information and realign authors with their current employer.