Research Capacity Building Workshops Edge Hill University
Author Citation Metrics h-index and Altmetrics Costas Gabrielatos
[email protected] 16 November 2016
Main focus h-index (Hirsch, 2005) Why? Used by the three indexing databases usually consulted by universities for purposes of evaluation and promotion: • Web of Knowledge (Thomson Reuters) • Scopus (Elsevier) • Google Scholar Citations (Google)
Contextualisation: REF (1) Time window: 5-6 years. Quantity: min. 1 - max. 4 publications.
Quality: expert review* • Relatively small number of reviewers. • Reviewers not always experts in subject matter or methodology. • Time not enough for in-depth evaluation. • Reviews are not blind.
* http://blogs.lse.ac.uk/impactofsocialsciences/2014/11/19/
Contextualisation: REF (2) “The government’s green paper on higher education reform provides the strongest signal yet that metrics are likely to play a more significant role in the next Research Excellence Framework. The next REF, to take place in 2021, will be based on peer review, according to the green paper Fulfilling our Potential: Teaching excellence, social mobility and student choice, which was published at one minute after midnight on 6 November. However, “We should champion better and more effective use of data and metrics,” the government says. It says it proposes to do this by bringing together and linking “existing, often fragmented and partial, databases that seek to go some way in capturing different measures of research excellence”. (Gallardo, 2015)
Contextualisation: REF (3) After REF, publications become ‘useless’. Publications not included in REF are ‘wasted’. REF system does not allow for output fluctuations. Expert reviews are influenced by … … author / journal / publisher reputation. … reviewer’s familiarity with topic. … reviewer’s theoretical / methodological views.
Proposed utility of h-index Evaluation of a researcher’s scholarly impact based on their entire output. It is designed to reflect both quantity and quality of output.
• Quantity is measured directly, by taking into account the number of publications. • Quality is measured indirectly, by taking into account the number of citations (taken to reflect the extent of perceived usefulness).
h-index: Definition “A scientist has index h if h of his/her Np papers have at least h citations each, and the other (Np-h) papers have no more than h citations each” (Hirsch, 2005). Example An h-index of 5 means that 5 of the researcher’s papers have at least 5 citations each – and the rest have 5 or fewer.
Characteristics: Indexing (1) For papers to be included in the calculation of a researcher’s h-index … … the cited papers must be indexed in a database and …the citing papers must be in publications indexed in the same database.
Characteristics: Indexing (2) • The most widely used databases predominantly contain: • Publications reporting on original research. • Journal papers. • STEM proceedings • Edited volumes (e.g. proceedings, handbooks) are rarely indexed. • Monographs are not included. • Exception: state of the art papers/chapters.
Characteristics: Indexing (3) More inclusive databases tend to return higher h-index scores.
The same paper may have a different number of citations in different databases. The same researcher may have different h-index scores in different databases. The type of publication is important. The place of publication is important. The particular database is important.
Characteristics: Outputs+Citations (1) The number of a researcher’s indexed publications indicate their maximum possible h-index score. •
E.g., a researcher with 5 indexed papers cannot have an h-index higher than 5 – even if each paper has more than five citations.
For the h-index to go up, … • the researcher needs to publish more papers, and/or • the researcher’s published papers (old and new) need to attract more citations.
Characteristics: Outputs+Citations (2) The longer a researcher has been publishing, the higher their h-index is expected to be. Early-career researchers can be expected to have (very) low h-index scores.
The h-index is not very good at indicating emerging/promising academics.
Characteristics: Outputs+Citations (3) The h-index minimizes the influence of a few very highly cited papers. Paper Paper A Paper B Paper C Paper D Paper E Paper F Paper G
Citations 58 32 6 4 3 2 1
h-index: 4
Characteristics: Outputs+Citations (4) Usually, only a small sub-set of a researcher’s cited papers is represented in their h-index
Prolific researchers do not necessarily have high h-index scores.
Characteristics: Citation Distribution (1) Two hypothetical researchers (R1, R2) Both have published 10 indexed papers. Both have a total of 100 citations. Does it mean they have the same h-index score?
Characteristics: Citation Distribution (1) Papers
Citations (N=100) R1
Paper A
34
Paper B Paper C Paper D Paper E Paper F Paper G Paper H Paper I Paper J
21 17 12 5 4 3 2 1 1
Researcher 1 h-index: 5
Characteristics: Citation Distribution (1) Papers
Citations (N=100) R1 R2
Paper A
34
10
Paper B Paper C Paper D Paper E Paper F Paper G Paper H Paper I Paper J
21 c 17 12 5 4 3 2 1 1
10 10 10 10 10 10 10 10 10
Researcher 1 h-index: 5
Researcher 2 h-index: 10
Characteristics: Citation Distribution (2) Two hypothetical researchers (R1, R2) Both have published 7 indexed papers. R1 has 106 citations; R2 has 49 citations. Does it mean R1 has a higher h-index score?
Characteristics: Citation Distribution (2) Papers
Citations R1 R2 (N=106)
(N=49)
Paper A
58
7
Paper B
32
7
c
Paper C
6
7
Paper D
4
7
Paper E
3
7
Paper F
2
7
Paper G
1
7
R1 Citations: 106 h-index: 4
R2 Citations: 49 h-index: 7
Characteristics: Citation Distribution (3) The distribution of citations is as important as their total number -- and in some cases, more important. Two researchers with the same number of citations may have different h-index scores. Two researchers with different number of citations may have the same h-index score. A researcher may even have a lower h-index score than a researcher with fewer citations.
Characteristics: Time Time-lag between publication and citations. It usually takes 2-4 years from publication for citations to materialize.
Even with an initial burst of publications, it will take early-career researchers a fairly long time to develop a competitive h-index score.
Characteristics: Researcher ‘Brand’ A researcher’s popularity/network are major factors contributing to the number of citations they accumulate -- particularly given the proliferation of publications.
Established researchers have an advantage over early- or mid-career ones. Need for researchers to be visible: Networking, Conferences, Web Presence
Characteristics: Subject Matter Different (sub-)fields have different … • output volumes • authorship practices • publication practices • citation practices
Comparisons across fields, even sub-fields, are bound to be misleading.
Differences in citation rates between major groups of disciplines (citation rate = total citations divided by number of publications)
http://blogs.lse.ac.uk/impactofsocialsciences/the-handbook/chapter-1-what-shapes-the-citing-of-academic-publications
“There are many possible reasons for this patterning. In medicine all published papers are written to a word limit of 3,000 words, whereas the norm in the social sciences is for main papers to be around 6,000 to 9,000 words long. Medical sciences have also developed a strong and rigorous culture of ‘systematic review’ which requires that all relevant studies be cited initially, but that only those that pass certain criteria for methods and merit need be analysed closely. This very structured and well-defined approach to reviewing literature is mirrored (perhaps in a less rigorous way) in the physical sciences. But a culture of systematic review or comprehensive referencing is far from being established in most social science disciplines – for instance, in theoretical economics and public choice only methodologically similar work is cited, and authors often make a cult of minimal referencing. Systematic review, or a stress on comprehensive referencing, is entirely absent in the humanities.” http://blogs.lse.ac.uk/impactofsocialsciences/the-handbook/chapter-1-what-shapes-the-citing-of-academic-publications
Improvements on the h-index (1) There is a growing number of citation indices aiming to address the limitations of h-index. They take account of one or more of the following: • citations to all publications • age of publication • highly cited publications • rate of citation accumulation • number of authors • relative contribution of authors • self-citations • citations in the same journal Sources: • http://www.harzing.com/pophelp/metrics.htm • http://googlescholar.blogspot.co.uk/2011/11
Improvements on the h-index (2) i-10 index • Number of publications with at least 10 citations. • Used by Google Scholar citations
Egghe's g-index • Gives more weight to highly-cited articles. Contemporary h-index • Gives more weight to recent articles Rewards academics with a steady output level. Age-weighted citation rate (AWCR) and AW-index • Average number of citations to an entire body of work, adjusted for the age of each individual paper.
Improvements on the h-index (3) Zhang's e-index • The square root of the surplus in the h-set beyond the minimum required to obtain a h-index of 'h'. • Differentiates between scientists with similar scores but different citation patterns.
Average annual increase in the individual h-index • An indicator of an individual's average annual research impact. • Removes to a considerable extent disciplinespecific patterns that may distort the h-index. • Reduces career length effect provides a fairer comparison between junior and senior researchers.
Improvements on the h-index (5) Individual h-index • Divides the h-index by the average number of authors in the articles that contribute to it. • [Variant] Normalizes number of citations for each paper (dividing number of citations by number of authors for each paper), then calculates the hindex of the normalized citation counts. Multi-authored h-index • Uses fractional paper counts (according to number of authors), and then determines the multiauthored index based on the resulting effective rank of the papers using undiluted citation counts.
Alternative metrics (1) Altmetrics is new approach, which argues that citation metrics need to be supplemented with:
• Number of downloads from open-access repositories (e.g. E-prints, Academia.edu, Research Gate). • Number of bookmarks in citation managers (e.g. Mendeley, Zotero).
• Frequency of mentions (e.g. Twitter, Facebook).
in
social
media
Alternative metrics (2) Equate attention with use or usefulness. Favour researchers with large networks.
Make web presence imperative.
The Leiden Manifesto for Research Metrics (Hicks et al., 2015)
Principles guiding research evaluation (Hicks et al., 2015) • Base assessment of individual researchers on a qualitative judgement of their portfolio. • Quantitative evaluation should support qualitative, expert assessment. • Account for variation by field in publication and citation practices. • Avoid misplaced concreteness and false precision. • Recognize the systemic effects of assessment and indicators. • Scrutinize indicators regularly and update them. • Keep data collection and analytical processes open, transparent and simple. • Measure performance against the research missions of the institution, group or researcher. • Protect excellence in locally relevant research. • Allow those evaluated to verify data and analysis.
The Metric Tide Report of the Independent Review of the Role of Metrics in Research Assessment and Management (LSE, 2015) http://blogs.lse.ac.uk/impactofsocialsciences/files/ 2015/07/2015_metrictide.pdf
The Metric Tide: Conclusions (1) • There is considerable scepticism among researchers, universities, representative bodies and learned societies about the broader use of metrics in research assessment and management. • Peer review, despite its flaws, continues to command widespread support as the primary basis for evaluating research outputs, proposals and individuals. • However, a significant minority are enthusiastic about greater use of metrics, provided appropriate care is taken. • Carefully selected indicators can complement decision-making, but a ‘variable geometry’ of expert judgement, quantitative indicators and qualitative measures that respect research diversity will be required. • There is legitimate concern that some indicators can be misused or ‘gamed’: journal impact factors, university rankings and citation counts being three prominent examples.
The Metric Tide: Conclusions (2) • The data infrastructure that underpins the use of metrics and information about research remains fragmented, with insufficient interoperability between systems. • Analysis concluded that that no metric can currently provide a like-for-like replacement for REF peer review. • In assessing research outputs in the REF, it is not currently feasible to assess research outputs or impacts in the REF using quantitative indicators alone. • In assessing impact in the REF, it is not currently feasible to use quantitative indicators in place of narrative case studies. However, there is scope to enhance the use of data in assessing research environments.
The Metric Tide: Recommendations • Robustness: basing metrics on the best possible data in terms of accuracy and scope. • Humility: recognising that quantitative evaluation should support – but not supplant – qualitative, expert assessment. • Transparency: keeping data collection and analytical processes open and transparent, so that those being evaluated can test and verify the results. • Diversity: accounting for variation by field, and using a range of indicators to reflect and support a plurality of research and researcher career paths across the system. • Reflexivity: recognising and anticipating the systemic and potential effects of indicators, and updating them in response.
Attracting Citations Content
• • • •
Type
• Article, chapter, monograph, edited volume? • Indexed? Ranking?
Visibility
• Number of webpages where the paper is listed. • Number of mentions / recommendations in social media.
Research or pedagogical? Topic/focus? Methodology? Results?
• Open-access or paywalled? • If OA, gold or green? Accessibility • Number of webpages the paper can be downloaded from?
Citation Metrics: An example e.g. Noam Chomsky
• Google Scholar Citations • Web of Knowledge
• Scopus
Accessing your citation metrics Web of Knowledge • http://wok.mimas.ac.uk.edgehill.idm.oclc.org
Scopus • https://www.scopus.com/freelookup/form/author.uri
Google Scholar Citations • https://scholar.google.co.uk/intl/en/scholar/citations.html
References and more information (1) • Finch, A. (2010) Can we do better than existing author citation metrics? BioEssays, 32(9). [e-copy] • Gallardo, C. (2015) Metrics-based REF looks more likely. Research Professional, 06-11-2015. [e-copy] • Harzing, A.W. (2007) Publish or Perish. [e-copy] • Hicks, D., Wouters, P., Waltman, L., de Rijcke, S. & Rafols, I. (2015). The Leiden Manifesto for research metrics. Nature, 520, 429-431. [e-copy] • Hirsch, J.E. (2005) An index to quantify an individual's scientific research output. [e-copy] • LSE Public Policy Group (2011) Maximising the Impacts of Your Research: A Handbook for Social Scientists. [e-copy]
References and more information (2) • University of California, Irving. (nd) Research Impacts Using Citation Metrics. [e-copy] • A basketful of metrics. [e-copy] • Altmetrics: A manifesto. [e-copy] • Citation Metrics (Publish or Perish 4 User's Manual). [e-copy] • Metrics for All? HEFCE [e-copy] • Metrics: How to handle them responsibly. Times Higher Education. [e-copy] • One scholar’s crusade against the REF. Times Higher Education. [e-copy]
References and more information (3) • Slouching toward the Market: The new Green Paper for Higher Education. Part I, Part II • The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. HEFCE. [ecopy] • The usefulness of citation counts depends heavily on the context of an individual’s publishing community. LSE Impact Blog. [e-copy] • Time to abandon the gold standard? Peer review for the REF falls far short of internationally accepted standards. LSE Impact Blog. [e-copy] • Use of metrics threatens researcher diversity. Research Professional (2014) [e-copy]
