[Purpose/Significance] This paper analyzes the difference of institute impact evalution based on different subject classification systems and their impact on the evaluation results. [Method/Process] The study selected 5 classification systems, and 8 classification schemes from Incites database. Firstly, it conducted correlation analysis on CNCI of 14,955 institutions based on different classification schemes to study the similariy of the evaluation results under different classification systems. Then it selected 36 double first-class universities in China, compared the changes of CNCI values based on different classification schemes, analyzed the specific reasons for the differences, and studied the effect of classification system on the evaluation of individual institutions. [Result/Conclusion] The CNCI values obtained under different subject classification schemes were significantly correlated (the lowest correlation reached 0.85).This indicates that the evaluation results obtained by different classification systems have high similarity. However, the evaluation results under different classification systems also have clustering characteristics. OECD, ESI, SCADC and CT1 have high correlation coefficients and more similar results, and WoS, CT2 and CT3 evaluation results are more similar. The granularity of classification system is an important factor to determine the evaluation result. The overall correlation of the evaluation results of 36 universities under different classification systems is high, but the CNCI values of some universities vary greatly, especially those institutions with prominent publications on hot topics. The fundamental reason for the great difference in the evaluation results is that the same paper is divided into different categories and the citation baselines under different categories are different. In the evaluation process, more fine-grained classification system is recommended to reduce the influence of hot topics on the citation baselines.
Geng Haiying
,
Yang Liying
,
Shen Zhesi
. The Impact of Different Subject Classification Systems on the Evaluation of Scientific Research Impact of Institutes[J]. Library and Information Service, 2022
, 66(23)
: 104
-114
.
DOI: 10.13266/j.issn.0252-3116.2022.23.011
[1] LEYDESDORFF L, BORNMANN L, ZHOU P. Construction of a pragmatic base line for journal classifications and maps based on aggregated journal-journal citation relations[J]. Journal of informetrics, 2016,10(4):902-918.
[2] HAUNSCHILD R, SCHIER H, MARX W, et al. Algorithmically generated subject categories based on citation relations: an empirical micro study using papers on overall water splitting[J]. Journal of informetrics, 2018,12(2):436-447.
[3] WANG Q, WALTMAN L. Large-scale analysis of the accuracy of the journal classification systems of Web of Science and Scopus[J]. Journal of informetrics, 2016,10(2):347-364.
[4] ZITT M, LELU A, CADOT M, et al. Bibliometric delineation of scientific fields[M]//GLANZEL W, MOED H F, SCHMOCH U, et al. Springer handbooks. Cham: Springer, 2019:25-68.
[5] 张蕊, 祖文玲, 邱均平, 等. 2021-2022世界一流大学与一流学科评价及结果分析[J]. 评价与管理, 2021,19(4):67-72.
[6] 中国科研国际竞争力研究课题组. 中国基础研究国际竞争力蓝皮书2015[M].北京:科学出版社.2015.
[7] 软科. 软科排名[EB/OL]. [2022-09-23]. https://www.shanghairanking.cn/.
[8] LEYDESDORFF L, BORNMANN L. The operationalization of "fields" as WoS subject categories (WCs) in evaluative bibliometrics: the cases of "library and information science" and "science & technology studies"[J]. Journal of the Association for Information Science and Technology, 2016,67(3):707-714.
[9] 邱均平, 舒非, 卢坚, 等. 面向评价需求的ESI学科分类与我国一流学科类目的匹配研究[J]. 重庆大学学报(社会科学版), 2021,27(1):110-121.
[10] GLANZEL W, SCHUBERT A. A new classification scheme of science fields and subfields designed for scientometric evaluation purposes[J]. Scientometrics, 2003,56(3):357-367.
[11] ARCHAMBAULT É, BEAUCHESNE O H, CARUSO J. Towards a multilingual, comprehensive and open scientific journal ontology[EB/OL]. [2022-09-20]http://www.science-metrix.com/pdf/ Towards a Multilingual Comprehensive and Open.
[12] BOERNER K, KLAVANS R, PATEK M, et al. Design and update of a classification system: the UCSD map of science[J]. Plos one, 2012,7(e394647):1-10.
[13] WALTMAN L, VAN ECK N J. A new methodology for constructing a publication-level classification system of science[J]. Journal of the American Society for Information Science and Technology, 2012,63(12):2378-2392.
[14] KLAVANS R, BOYACK K W. Which type of citation analysis generates the most accurate taxonomy of scientific and technical knowledge?[J]. Journal of the Association for Information Science and Technology, 2017,68(4):984-998.
[15] PERIANES-RODRIGUEZ A, RUIZ-CASTILLO J. A comparison of the Web of Science and publication-level classification systems of science[J]. Journal of informetrics, 2017,11(1):32-45.
[16] 任元秋, 王兴, 郑钦钦. 不同学科分类方案下不同学科标准化方法效果的比较研究[J]. 图书情报工作, 2021,65(3):84-92.
[17] RUIZ-CASTILLO J, WALTMAN L. Field-normalized citation impact indicators using algorithmically constructed classification systems of science[J]. Journal of informetrics, 2015,9(1):102-117.
[18] HAUNSCHILD R, MARX W, FRENCH B, et al. Relationship between field-normalized indicators calculated with different approaches of field-categorization[C]//STI 2018 conference proceedings. Leiden University:Centre for Science and Technology Studies,2018:11-18.
[19] SZOMSZOR M, ADAMS J, PENDLEBURY D A, et al. Data categorization: understanding choices and outcomes[EB/OL].[2022-09-15].https://ccn.unistra.fr/websites/ccn/documentation/Recherche-Data/ISI-GRR-2021-Data-categorization-understanding-choices-and-outcomes.pdf.
[20] SILE L, GUNS R, VANDERMOERE F, et al. Tracing the context in disciplinary classifications: a bibliometric pairwise comparison of five classifications of journals in the social sciences and humanities[J]. Quantitative science studies, 2021,2(1):65-88.
[21] BENSMAN S J, LEYDESDORFF L. Definition and identification of journals as bibliographic and subject entities: librarianship versus ISI journal citation reports methods and their effect on citation measures[J]. Journal of the American Society for Information Science and Technology, 2009,60(6):1097-1117.
[22] PERIANES-RODRIGUEZ A, RUIZ-CASTILLO J. The impact of classification systems in the evaluation of the research performance of the Leiden Ranking universities[J]. Journal of the Association for Information Science and Technology, 2018,69(8):1046-1053.
[23] 美国新闻与世界报道. U.S.News大学排名[EB/OL]. [2022-09-10]. https://www.usnews.com/rankings.
[24] OECD. Frascati Manual 2015[EB/OL]. [2022-08-06]. https://www.oecd.org/sti/frascati-manual-2015-9789264239012-en.htm.
[25] 国务院学位委员会. 学位授予和人才培养学科目录(2018年4月更新)[EB/OL]. [2022-08-06]. http://www.moe.gov.cn/s78/A22/xwb_left/moe_833/201804/t20180419_333655.html.
[26] 国务院学位委员会, 教育部. 研究生教育学科专业目录管理办法[EB/OL]. [2022-09-15]. http://www.gov.cn/zhengce/zhengceku/2022-09/14/5709785/files/612f061295fb42668601c17a6524f753.pdf.
[27] PUDOVKIN A I, GARFIELD E. Algorithmic procedure for finding semantically related journals[J]. Journal of the American Society for Information Science and Technology, 2002,53(13):1113-1119.
