Scientometrics DOI 10.1007/s11192-010-0333-2
Effect on the journal impact factor of the number and document type of citing records: a wide-scale study Juan Miguel Campanario • Jesu´s Carretero • Vera Marangon Antonio Molina • Germa´n Ros
•
Received: 6 July 2010 Ó Akade´miai Kiado´, Budapest, Hungary 2010
Abstract We studied the effect on journal impact factors (JIF) of citations from documents labeled as articles and reviews (usually peer reviewed) versus citations coming from other documents. In addition, we studied the effect on JIF of the number of citing records. This number is usually different from the number of citations. We selected a set of 700 journals indexed in the SCI section of JCR that receive a low number of citations. The reason for this choice is that in these instances some citations may have a greater impact on the JIF than in more highly-cited journals. After excluding some journals for different reasons, our sample consisted of 674 journals. We obtained data on citations that contributed to the JIF for the years 1998–2006. In general, we found that most journals obtained citations that contribute to the impact factor from documents labeled as articles and reviews. In addition, in most of journals the ratio between citations that contributed to the impact factor and citing records was greater than 80% in all years. Thus, in general, we did not find evidence that citations that contributed to the impact factor were dependent on non-peer reviewed documents or only a few citing records. Keywords
Journal impact factor Citations Citing records
Introduction The journal impact factor (JIF) continues to be one of the most widely used scientometric indicators. It is computed for each year (Y) according to the following equation (Gla¨nzel and Moed 2002):
J. M. Campanario (&) J. Carretero A. Molina G. Ros Departamento de Fı´sica, Edificio de Ciencias, Universidad de Alcala´, 28871 Alcala´ de Henares, Madrid, Spain e-mail:
[email protected] V. Marangon Facultad de Ciencias Econo´micas, Universidad de Alcala´, 28807 Alcala´ de Henares, Madrid, Spain
123
J. M. Campanario et al.
JIF ðYÞ ¼
Citations in Y to documents published in Y1 and Y2 Citable items published in Y1 and Y2
where Y1 and Y2 are the 2 years before Y. To calculate the numerator of the JIF, ISI Thompson Reuters counts citations to all types of documents, whereas the denominator includes only citable documents (Gla¨nzel and Moed 2002; Golubic et al. 2008). A vast literature exists on the JIF (see, for example, a recent review by Archambault and Larivie`re (2009). In previous studies we introduced the concept of structure of the JIF. By ‘‘structure’’ we mean the different subsets used to classify citations (Campanario et al. 2006). The numerator of the JIF equation is a set of citations which can be grouped in subsets according to different criteria. In previous studies we examined the structure of the JIF of academic journals from different perspectives. For example, we previously calculated the percentage of citations that contributed to the JIF and that appeared in articles authored by members of the journal’s editorial board (Campanario et al. 2006). In other studies we analyzed the effect on JIFs of citations in documents labeled by Thompson Scientific as ‘‘editorial material’’ (Gonzalez and Campanario 2007; Campanario and Gonzalez 2006) and the possible use of journal self-citations to increase JIFs (Campanario and Molina 2009; Andrade et al. 2009). In general, we did not find evidence of wide-scale use of journal self-citations to increase the JIF of the journals we studied. Here we turn our attention to other factors that could contribute to the structure of JIFs. For example, consider the case of two journals that receive 10 citations that contribute to the JIF. Now consider that all citations to the first journal come from a single article, whereas citations to the second journal come from 10 different articles. We could think that the true ‘‘impact’’ of the two journals differs. It could be argued that the second journal has ‘‘had an impact on’’ more articles than the first one. This reasoning makes it interesting to explore differences among journals according to the number and type of citing records that contribute to the JIF. Consider now two journals that receive 10 citations that contribute to the JIF. Now suppose that the first journal is cited only in documents labeled by ISI Thomson Reuters as articles and reviews, whereas the second one is cited in documents labeled as editorial material, book review, letter, correction or some other type of document. The citing documents that cite the first journal are usually peer reviewed and are considered more relevant than the citing documents that cite the second journal. In most journals, articles and reviews are peer reviewed and published mainly because of their scientific merit. However, other documents can be published according to other criteria and goals. Actually, the idea of the JIF is closer to the notion of impact related to ‘‘scientific’’ origin. In fact, the denominator of the equation used to compute the JIF includes only articles and reviews. It seems to us that the contribution to the JIF of citations coming from documents other than articles and reviews could be counted in a different way. We should keep in mind that in journals whose JIF is based on a high number of citations, the influence of any few citations in this scientometric indicator is small. However, in journals that are not cited very much, the influence of individual citations in the computation of the JIF may be much greater. For example, citations coming from only one or two documents or citations coming from documents other than articles and reviews could boost the JIF to an extent greater than what would be expected for journals that receive many citations.
123
A wide-scale study
In light of the above considerations, the goals of this study were: (a)
To introduce a new approach to the study of the structure of JIFs in a wide set of journals. (b) To study the relative contribution to the JIF of the number of citing records. (c) To study the relative contributions to the JIF of citations coming from more researchoriented documents (articles and reviews) versus other types of citing documents.
Method We selected 700 journals from the JCR database (SCI). Only journals listed (short name) in JCR in all years from 1998 to 2006 were selected. For each journal, we calculated the sum of citations received each year, according to data published in the JCR database. Journals were ranked from lowest to highest based on the total number of citations received. Next, the first 700 journals listed in the SCI database were selected. For each journal, we searched all cited records with a publication year from 1996 to 2005. This range allowed us to study all JIFs from 1998 to 2006. We used the Web of Science (WOS) web interface available to universities in Spain, with the default limits (all years, all databases). The WOS screen showing the result of the Cited Reference Search shows a number of fields for each record. These fields are Select, Cited Author, Cited Work, Year, Volume, Page, Article ID, Citing Articles and View Record (see Box 1). For each cited record, we recorded all fields to obtain a Cited Reference field by combining the relevant fields (see Box 1 for an example). Next, we obtained all citing record citations to each cited reference. Records were downloaded from WOS without restrictions for year or any other characteristic. This allowed us to identify all citing records of each Cited Reference. We used a computer program to construct a file in which each cited reference was linked to all its citing records. We checked that for each Cited Reference, we obtained a number of citing records equal to the number that appeared in the Citing Articles field. In addition, we checked that each Cited Reference appeared as a text string in the CR field of their citing records. The CR field includes all references cited in a given citing record. This method allowed us to analyze all citations to any given cited reference. After we completed the above checks, we discarded all citing records with publication years outside the range studied here (1998–2006). To avoid spurious results, we deleted from our list all journals with more than one abbreviated title that could be used to search by cited records. For example, the journal Box 1 Example of a cited reference search Original record from the Cited Reference Search screen Select Cited author
ABDULLAH MZ
Cited work
Year Volume Page Article ID
T I MEAS CONTROL
2005 27
65
Citing articles
DOI 10.1191/ 2 0142331205 tm138oa
View record
View Record
Cited reference (constructed using relevant fields) ABDULLAH MZ, 2005, T I MEAS CONTROL, V27, P65, DOI 10.1191/0142331205tm138oa
123
J. M. Campanario et al.
Acta Zoologica Academiae Scientiarum Hungaricae is identified by ISI-Thomson Reuters both as ACTA ZOOL ACAD SCI H and ACTA ZOOL HUNG. The journals excluded for of this reason were ACTA ZOOL ACAD SCI H, ADV IMAG ELECT PHYS, ADV POWDER TECHNOL, ASHRAE J, COMPUTATION STAT, FOOD AGR IMMUNOL, FOOD BIOTECHNOL, POSITIVITY, REAL-TIME IMAGING, STOCH ANAL APPL, THEOR FOUND CHEM ENG and ZKG INT. A similar problem affected IEE REV, as explained to us by ISI-Thomson Reuters technical support staff. Changes were made in the abbreviations of this journal but the changes were not implemented in all records. We also excluded other journals because of problems with the downloading process or citation analysis: AUST NZ J STAT, MAR TECHNOL SOC J, ARCH FISH MAR RES, ICHTHYOL RES, CIENC MAR (some citing records were empty); VERHALTENSTHERAPIE (this journal is included in both SCI and SSCI), FIBER INTEGRATED OPT, NEURAL PROCESS LETT, ENVIRON ENG SCI, COMBUST THEOR MODEL; J THEOR PROBAB (some cited records were cited 0 times); INDIAN J PURE AP MAT (we were unable to process the downloads because of problems caused by citations to a Cited Reference). For one journal we were unable to find citations that contributed to the impact factor. This journal is SADHANA-ACAD P ENG S. After these journals were excluded, the final sample consisted of 674 journals. Our final file contained a long list of records that each included the data of one cited reference and one citing document. We discarded all records in which the publication type was ‘‘C’’ (Congress). In recent years, ISI-Thomson Reuters has started to include meeting and congress proceedings as a source of records, but we were interested in data from citing journals only. Citations indexed by ISI-Thomson Reuters as from a congress may not have been counted in the calculation of the JIF. For example, if the proceedings of a given congress were indexed recently, citations from these proceedings were probably not counted in the computation of the JIF of cited journals. Moreover, documents published in journals tend to be considered more valuable than items published in meeting and congress proceedings. In addition, in most fields, the peer review process for journals is generally more stringent that the peer review process for congresses and meetings. However, in some fields (computer science and physics the peer review process can be also stringent for congresses and meetings. We then selected only records corresponding to citations that contributed to the JIF. These are cited references for which the citing document was published in year Y (for example, 2004) and the cited reference was published in year Y-1 (2003) or Y-2 (2002). These records were classified in two categories: (a)
Citing documents labeled by ISI-Thomson Reuters as an article or review (we labeled these citations AR) (b) Citing documents labeled by ISI-Thomson Reuters as other kinds of document (we labeled these citations OT). We did not try to reproduce the number of citations used by ISI-Thomson Reuters to compute the JIF. It is well know that the number of citations used by ISI-Thomson Reuters to compute the JIF does not always match the number of citations downloaded with the WOS data. For example, ISI-Thomson Reuters staff told us that when computing the JIF, they sometimes use manual procedures to delete or add citations (Campanario et al. 2006). Other authors have noted that the total number of citations extracted from WOS data is different from the number listed by the ISI in the Journal Citations Reports (Christensen et al. 1997; Rossner et al. 2007, 2008). Thus, we consider our data as a proxy for the number of citations that contributed to the JIF.
123
A wide-scale study
For each journal and year studied we calculated the following variables: CIF: Number of citations that contributed to the JIF. CIF-AR: Number of citations that contributed to the JIF coming from documents labeled articles or reviews by ISI-Thomson Reuters. CIF-OT: Citations that contributed to the JIF coming from other documents. PCIF-AR and PCIF-OT: The variables CIF-AR and CIF-OT divided by CIF, with the quotient expressed as a percentage. CR/CIF: Number of citing records divided by citations that contributed to the JIF (CIF), expressed as a percentage. This variable has a maximum value of 100 when the number of citations and the number of citing documents are the same. If CIF is much greater than the number of citing records, this would suggest that JIF is mainly dependent on citations received from a small number of records. These variables allowed us to study the distribution of journals according the contribution to the JIF of citations from articles and reviews versus citations from other documents. We also studied the year-to-year evolution of the contribution of citations from other documents and the effect of the number of citing records on CIF. We did not try to compare groups, so no statistical tests for significance were required. Our basic approach was to study the relationships between changes in the variables. Results Distribution of journals in SCI groups Journals are indexed in JCR in groups. Many journals are indexed in two or more groups. Table 1 shows the groups that appear more frequently in the set of journals we studied. Engineering and Biological Sciences are very common in the list. Distribution of journals according the percentage of articles that they publish Table 2 shows the distribution of journals according to the percentage of documents labeled by ISI as ‘‘articles’’ that they publish. Most of journals studied are publishing research articles. Distribution of journals according to the contribution to JIF of citations from articles and reviews and other documents Table 3 show, for each year, the distribution of journals according the value of PCIF-AR. In each set, distributions are very similar for all years studied. Citations from documents other than articles or reviews made relevant contributions to the CIF in only a few journals. Increase in PCIF-OT with time We studied the increase in PCIF-OT from one year to the next. Table 4 shows the distribution of journals according to the number of increments in the variable PCIF-OT. Almost 50% of journals showed only 1 or 2 yearly increases. About 35% of journals had 3 or more yearly increments. However, no clear trend was evident toward large, sustained increases in the contribution of citations from other documents to the impact factor (CIF). We found only 27 journals with three or more consecutive increments in PCIF-OT.
123
J. M. Campanario et al. Table 1 JCR groups in which journals are indexed
Only groups that appear more than 10 times are included. Note that some journals are indexed in more than one group
Group
Times
Engineering, Electrical & Electronic
49
Veterinary Sciences
31
Materials Science, Multidisciplinary
29
Mathematics
26
Engineering, Mechanical
25
History & Philosophy of Science
21
Computer Science, Theory & Methods
20
Computer Science, Artificial Intelligence
20
Engineering, Multidisciplinary
19
Engineering, Civil
19
Plant Sciences
18
Engineering, Chemical
18
Zoology
16
Telecommunications
15
Food Science & Technology
15
Computer Science, Interdisciplinary Applications
15
Biotechnology & Applied Microbiology
15
Computer Science, Information Systems
15
Sport Sciences
14
Multidisciplinary Sciences
14
Mechanics
14
Mathematics, Applied
14
Computer Science, Hardware & Architecture
14
Neurosciences
13
Chemistry, Multidisciplinary
13
Biochemistry & Molecular Biology
12
Polymer Science
12
Operations Research & Management Science
12
Metallurgy & Metallurgical Engineering
11
Medicine, General & Internal
11
Instruments & Instrumentation
11
Entomology
11
Distribution of journals with values of PCIF-OT higher than 30% Table 5 shows the distribution of journals with a value of PCIF-OT higher than 30% in different years. In only a few journals the value of CIF was related to a significant percentage of citations from documents other than articles or reviews in at least 3 years. In these journals the JIF depended mostly on citations from documents that, in general, were not peer reviewed. These journals are indexed in groups related mainly to engineering and medical sciences. Distribution of journals according to the variable CR/CIF Table 6 shows the distribution of journals according the value of CR/CIF in each year. The distributions were similar for all years. In all years, CR/CIF was greater than 80% in about
123
A wide-scale study Table 2 Distribution of journals according the percentage of document labeled by ISI as ‘‘articles’’ they publish
Percent of articles
N Journals
%
10
5
0.7
20
18
2.7
30
22
3.3
40
19
2.8
50
30
4.5
60
52
7.7
70
67
9.9
80
104
15.4
90
153
22.7
100
204
30.3
Total
674
100.0
Table 3 Distribution of journals according to the value of PCIF-AR Percentage
Year 1998
1999
2000
2001
2002
2003
2004
2005
2006
0–10
0.5
0.2
0.2
0.0
0.6
0.0
0.6
0.0
0.0
11–20
0.0
0.0
0.0
0.2
0.0
0.0
0.0
0.0
0.0
21–30
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.2
31–40
0.2
0.0
0.3
0.0
0.2
0.2
0.0
0.0
0.0
41–50
0.8
0.6
0.6
0.5
0.5
0.8
0.6
0.3
0.3
51–60
0.8
0.5
0.5
0.3
0.3
0.0
0.2
0.6
0.3
61–70
1.5
1.2
1.8
1.2
1.7
2.9
1.5
0.8
0.6
71–80
3.7
2.3
3.3
4.0
4.0
2.4
3.5
2.6
3.2
81–90
10.8
10.4
10.9
9.0
9.6
10.6
9.4
8.6
8.3
91–100
81.7
84.9
82.5
84.9
83.3
83.3
84.3
87.2
87.2
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
Table 4 Distribution of journals according the number of increments in the variable PCIF-OT
Number of increases
Number of journals
%
0
99
14.7
1
173
25.7
2
163
24.2
3
133
19.7
4
87
12.9
5
16
2.4
6
3
0.4
7
0
0.0
8
0
0.0
674
100.0
Total
123
J. M. Campanario et al. Table 5 Distribution of journals with a PCIF-OT higher than 30% in different years
Number of years
Number of journals
0
570
84.6
1
74
11.0
2
19
2.8
3
6
0.9
4
2
0.3
5
3
0.4
6
0
0.0
7
0
0.0
8
0
0.0
674
100.0
Total
%
Table 6 Distribution of journals according to the value of CR/CIF Percentage
Year 1998
1999
2000
2001
2002
2003
2004
2005
2006
0–10
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
11–20
0.2
0.0
0.0
0.0
0.0
0.3
0.0
0.0
0.2
21–30
0.2
0.0
0.0
0.2
0.0
0.0
0.2
0.2
0.3
31–40
0.2
0.2
0.3
0.3
0.6
0.6
1.2
0.3
0.5
41–50
0.6
0.6
1.2
0.9
1.4
0.5
0.9
1.2
0.6
51–60
2.2
2.1
1.7
3.2
2.3
0.9
1.5
1.5
1.8
61–70
5.1
6.3
2.0
5.1
5.3
4.7
3.2
4.2
5.0
71–80
10.9
11.2
11.7
10.9
12.0
11.0
10.8
13.3
9.4
81–90
22.2
21.7
26.5
21.1
22.0
22.1
22.7
23.8
24.9
91–100
58.5
57.9
56.7
58.3
56.4
59.9
59.5
55.5
57.5
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
90% of journals. This means that citations were not too concentrated in a given set of citing records. Changes in CR/CIF with time We studied the decrease in CR/CIF from 1 year to the next. This variable represents an increase in the number of citations that contributed to the impact factor (CIF) per citing record. Table 7 shows the distribution of journals according to the number of year-to-year decreases in CR/CIF. The most frequent number of yearly decreases was 3. There was no clear trend toward large, sustained increases in CR/CIF with time. Distribution of journals with values of CR/CIF lower than 70% For each year, we identified journals that had a value of CR/CIF lower than 70%. Table 8 shows the distribution of journals with a given value of CR/CIF in different years. Only 26
123
A wide-scale study Table 7 Distribution of journals according the number of decreases in the variable CR/CIF
Number of decreases
Number of journals
0
11
1
61
9.1
2
126
18.7
3
208
30.9
4
182
27.0
5
72
10.7
6
14
2.1
7
0
0.0
8 Total
Table 8 Distribution of journals in which the value of CR/CIF was lower than 70% during 3 or more years
% 1.6
0
0.0
674
100.0
Number of years
Number of journals
0
375
55.6
1
157
23.3
2
60
8.9
3
36
5.3
4
20
3.0
5
19
2.8
6
5
0.7
7
1
0.1
8
1
0.1
674
100.0
Total
%
or 674 journals had a CR/CIF value lower than 70% during 5 years or more. These journals are indexed in groups related mainly to engineering and life sciences.
Conclusions The JIF has a great influence on researchers’ publication strategies and decisions. The study of the structure of JIF should be a strategic research area that could help us to understand the nature and evolution of scientific impact. Our research suggests a new perspective on the study of the JIF, and is potentially useful to study the dynamics of science. We found that in most journals, the JIF depended on citations coming from documents labeled by ISI-Thomson Reuters as articles or reviews. As noted above, these documents are usually peer reviewed. Our results suggest that the JIF was not greatly influenced by documents that usually are not peer reviewed (for example, editorial material and other article types). We detected no large changes with time in the sample of journals we studied. In addition, we found that for most journals, the number of citing records that contributed to the JIF was similar to the number of citations that contributed to the impact
123
J. M. Campanario et al.
factor (CIF). There were no large changes in this pattern during the period from 1998 to 2006. Because our sample included journals that were not frequently cited, any change in the origin of citations from a small pool of citing records would have been expected to strongly influence the JIF. Acknowledgments This work was supported by a grant from the Spanish Ministry of Science and Technology (Direccio´n General de Investigacio´n) and the European Regional Development Fund (ERDF/ FEDER, project SEJ2007-66236/SOCI). We thank K. Shashok for improving the use of English in the manuscript and for suggestions about the content. We also thank an anonymous referee for suggestions.
References Andrade, A., Gonza´lez-Jonte, R., & Campanario, J. M. (2009). Journals that increase their impact factor at least fourfold in a few years: The role of journal self-citations. Scientometrics, 80(2), 515–528. Archambault, E., & Larivie`re, V. (2009). History of the journal impact factor: Contingencies and consequences. Scientometrics, 79, 635–649. Campanario, J. M., & Gonzalez, L. (2006). Journal self-citations that contribute to the impact factor: Documents labeled ‘‘editorial material’’ in journals covered by the Science Citation Index. Scientometrics, 69, 365–386. Campanario, J. M., Gonzalez, L., & Rodriguez, C. (2006). Structure of the impact factor of academic journals in the field of Education and Educational Psychology: Citations from editorial board members. Scientometrics, 69, 37–56. Campanario, J. M., & Molina, A. (2009). Surviving bad times: The role of citations, self-citations and numbers of citable items in recovery of the journal impact factor after at least four years of continuous decreases. Scientometrics, 81(3), 859–864. Christensen, F. H., Ingwersen, P., & Wormell, I. (1997). Online determination of the journal impact factor and its international properties. Scientometrics, 40, 529–540. Gla¨nzel, W., & Moed, H. F. (2002). Journal impact measures in bibliometric research. Scientometrics, 53, 171–193. Golubic, R., Rudes, M., Kovacic, N., Marusic, M., & Marusic, A. (2008). Calculating impact factor: How bibliographical classification of journal items affects the impact factor of large and small journals. Science and Engineering Ethics, 14, 41–49. Gonzalez, L., & Campanario, J. M. (2007). Structure of the impact factor of journals included in the Social Sciences Citation Index: Citations from documents labeled ‘‘editorial material’’. Journal of the American Society for Information Science and Technology, 58, 252–262. Rossner, M., van Epps, H., & Hill, E. (2007). Show me the data. Journal of Cell Biology, 179, 1091–1092. Rossner, M., van Epps, H., & Hill, E. (2008). Irreproducible results: a response to Thomson Scientific. Journal of Cell Biology, 180, 254–255.
123
