RESEARCH PAPERS

Study on the Relationship Between Contribution Types and Impact Metrics

  • Gou Zhenyu ,
  • Han Yuxin ,
  • Zhang Lin ,
  • Huang Ying
Expand
  • 1 School of Information Management, Wuhan University, Wuhan 430072;
    2 Center for Science, Technology & Education Assessment (CSTEA), Wuhan University, Wuhan 430072
Gou Zhenyu, Doctoral Candidate; Han Yuxin, Doctroal Candidate; Zhang Lin, Professor, PhD, Doctoral Supervisor, Corresponding Author, E-mail: linzhang1117@whu.edu.cn; Huang Ying, Associate Professor, PhD, Doctoral Supervisor.

Received date: 2024-12-09

  Revised date: 2025-02-16

  Online published: 2025-11-11

Supported by

This work is supported by the general program of National Natural Science Foundation of China titled “From Indicators to Indicating:outcome Classification, Collaboration Patterns, and Impact Diffusion of Interdisciplinary Research”(Grant No. 72374160).

Abstract

[Purpose/Significance] Exploring the relationship between contribution types and impact metrics can provide a reference for the proper use of citation metrics and optimizing evaluation methods. [Method/Process] This study constructed a framework for classifying paper contributions and employed a BERT model to identify contribution types. Using psychology papers as an example, it analyzed the relationship between contribution types and their impact based on academic influence and altmetric impact. [Result/Conclusion] Significant differences exist in the citation impact and altmetric impact of papers with different contribution types. Papers contributed on theoretical advancements show higher citation rates and greater diversity of citing disciplines, but moderate altmetric impact. In contrast, papers contributed on “experiment” have the lowest citation rates and the weakest diversity of citing disciplines, yet demonstrate stronger altmetric impact. The results suggest that when using impact metrics to evaluate papers, it is essential to consider intrinsic attributes such as research contribution. A classification-based evaluation approach can enhance the scientific validity and comparability of the results.

Cite this article

Gou Zhenyu , Han Yuxin , Zhang Lin , Huang Ying . Study on the Relationship Between Contribution Types and Impact Metrics[J]. Library and Information Service, 2025 , 69(21) : 120 -130 . DOI: 10.13266/j.issn.0252-3116.2025.21.010

References

[1] 中华人民共和国科学技术部.关于开展科技人才评价改革试点的工作方案[EB/OL].[2025-04-16]. https://www.most.gov.cn/xxgk/xinxifenlei/fdzdgknr/qtwj/qtwj2022/202211/t20221109_183356.html.(Ministry of Science and Technology of the People's Republic of China. Work plan for pilot reform of science and technology talent e-valuation[EB/OL].[2025-04-16]. https://www.most.gov.cn/xxgk/xinxifenlei/fdzdgknr/qtwj/qtwj2022/202211/t20221109_183356.html.)
[2] 中华人民共和国教育部.关于规范高等学校SCI论文相关指标使用树立正确评价导向的若干意见[EB/OL].[2025-04-16]. http://www.moe.gov.cn/srcsite/A16/moe_784/202002/t20200223_423334.html.(Ministry of Education of the People's Republic of China. Several opinions on standardizing the use of indicators related to SCI papers in higher education institutions and establishing a correct evaluation orientation[EB/OL].[2025-04-16]. http://www.moe.gov.cn/srcsite/A16/moe_784/202002/t20200223_423334.html.)
[3] SARAH M. Progressing research impact assessment:a'contributions'approach[J]. Research evaluation, 2015, 24(4):405-419.
[4] 梁琪奇,郭凤娇,王婧菲,等.单篇学术论文创新类型研究[J].情报理论与实践, 2024, 47(9):166-173.(LIANG Q Q, GUO F J, WANG J F, et al. Research on individual academic paper innovation types[J]. Information studies:theory&application,2024, 47(9):166-173.)
[5] 周海晨,郑德俊,郦天宇.学术全文本的学术创新贡献识别探索[J].情报学报, 2020, 39(8):845-851.(ZHOU H C,ZHENG D J, LI T Y. Research on the identification of academic innovation contributions of full academic texts[J]. Journal of the China Society for Scientific and Technical Information, 2020,39(8):845-851.)
[6] 索传军,于果鑫.学术论文研究亮点的语言学特征与分布规律研究[J].图书情报工作, 2020, 64(9):104-113.(SUO C J,YU G X. Exploration of the research "highlights" in academic papers[J]. Library and information service, 2020, 64(9):104-113.)
[7] XIE J, GONG K, LI J, et al. A probe into 66 factors which are possibly associated with the number of citations an article received[J]. Scientometrics, 2019, 119(3):1429-1454.
[8] ZHANG L, SIVERTSEN G, DU H, et al. Gender differences in the aims and impacts of research[J]. Scientometrics, 2021,126(11):8861-8886.
[9] 索传军,盖双双,周志超.认知计算--单篇学术论文评价的新视角[J].中国图书馆学报, 2018, 44(1):50-61.(SUO C J,GAI S S, ZHOU Z C. Cognitive computing:a new perspective for evaluating the individual academic paper[J]. Journal of library science in China, 2018, 44(1):50-61.)
[10] 罗卓然,王玉琦,钱佳佳,等.学术论文创新性评价研究综述[J].情报学报, 2021, 40(7):780-790.(LUO Z R, WANG Y Q, QIAN J J, et al. Research review on innovation evaluation of academic papers[J]. Journal of the China Society for Scientific and Technical Information, 2021, 40(7):780-790.)
[11] COSTANZO B P, SÁNCHEZ L E. Innovation in impact assessment theory and practice:how is it captured in the literature?[J]. Environmental impact assessment review, 2019,79:106289.
[12] CHEN H, NGUYEN H, ALGHAMDI A. Constructing a high-quality dataset for automated creation of summaries of fundamental contributions of research articles[J]. Scientometrics,2022, 127(12):7061-7075.
[13] 徐雷,张亚菲,叶均玲.科技文献创新内容的识别、组织与应用进展[J].情报学报, 2024, 43(2):237-250.(XU L,ZHANG Y F, YE J L. Advances in identification, organization,and application of innovative content in scientific literature[J].Journal of the China Society for Scientific and Technical Information, 2024, 43(2):237-250.)
[14] VOGT L, D'SOUZA J, STOCKER M, et al. Toward representing research contributions in scholarly knowledge graphs using knowledge graph cells[C]//Proceedings of the ACM/IEEE joint conference on digital libraries in 2020. New York:Association for Computing Machinery, 2020:107-116.
[15] 李贺,杜杏叶.基于知识元的学术论文内容创新性智能化评价研究[J].图书情报工作, 2020, 64(1):93-104.(LI H, DU X Y. Research on intelligent evaluation for the content innovation of academic papers[J]. Library and information service, 2020,64(1):93-104.)
[16] 罗卓然,蔡乐,钱佳佳,等.学术论文创新贡献句识别研究[J].图书情报工作, 2021, 65(12):93-100.(LUO Z R, CAI L, QIAN J J, et al. Research on the recognition of innovative contribution sentences of academic papers[J]. Library and information service,2021, 65(12):93-100.)
[17] 崔京艳,邓媛,李春梅,等.中医学科技论文创新性和科学性的识别与评价[J].中国医药导报, 2018, 15(5):172-176.(CUI J Y, DENG Y, LI C M, et al. Recognition and evaluation of the innovation and scientific nature of Chinese medical science and technology papers[J]. China medical herald, 2018, 15(5):172-176.)
[18] 曹树金,闫欣阳,张倩,等.中外情报学论文创新性特征研究[J].图书情报工作, 2020, 64(1):80-92.(CAO S J, YAN X Y,ZHANG Q. Research on characteristics of innovation in Chinese and international academic literature of information science[J].Library and information service, 2020, 64(1):80-92.)
[19] 蔡乐,罗卓然,陆伟.学术论文科研贡献类型自动识别研究[J].情报理论与实践, 2023, 46(6):168-175.(CAI L, LUO Z R,LU W. Research on automatic recognition of scientific research contribution types of academic papers[J]. Information studies:theory&application, 2023, 46(6):168-175.)
[20] D'SOUZA J, AUER S, PEDERSEN T. SemEval-2021 Task 11:NLPContributionGraph--structuring scholarly NLP contributions for a research knowledge graph[J]. arXiv preprint, 2021,arXiv:2106.07385.
[21] CHAO W, CHEN M, ZHOU X, et al. A joint framework for identifying the type and arguments of scientific contribution[J].Scientometrics, 2023, 128(6):3347-3376.
[22] 温浩.科技文摘创新点语义识别与分类方法研究[J].情报学报, 2019, 38(3):249-256.(WEN H. Semantic recognition and classification method of innovation points in scientific and technological abstracts[J]. Journal of the China Society for Scientific and Technical Information, 2019, 38(3):249-256.)
[23] 曹树金,闫颂.基于语义角色信息的科技论文创新段落定位及功能句识别方法研究--以中文情报学领域论文为例[J].情报理论与实践, 2022, 45(11):1-9, 20.(CAO S J, YAN S. Research on innovative paragraph positioning and functional sentence identification method of scientific and technical papers based on semantic role information:a case study of papers in the field of Chinese intelligence[J]. Information studies:theory&application, 2022, 45(11):1-9, 20.)
[24] REHMAN T, SANYAL D K, CHATTOPADHYAY S,et al. Automatic generation of research highlights from scientific[C]//2nd workshop on extraction and evaluation of knowledge entities from scientific documents (EEKE'21),collocated with JCDL. New York:Association for Computing Machinery, 2021:69-70.
[25] GUPTA K, AHMAD A, GHOSAL T, et al. A BERT-based sequential deep neural architecture to identify contribution statements and extract phrases for triplets from scientific publications[J]. International journal on digital libraries, 2024,25(4):1-28.
[26] 冯长根.一种自然而然的科技成果评价方法值得国家推广[J].中国人大, 2017(7):33-35.(FENG C G. A natural method for evaluating technological achievements is worth promoting by the country[J]. The People's Congress of China, 2017(7):33-35.)
[27] 章成志,李铮.基于学术论文全文的创新研究评价句抽取研究[J].数据分析与知识发现, 2019, 3(10):12-19.(ZHANG C Z, LI Z. Extracting sentences of research originality from full text academic articles[J]. Data analysis and knowledge discovery,2019, 3(10):12-19.)
[28] LE X, CHU J, DENG S, et al. CiteOpinion:evidence-based evaluation tool for academic contributions of research papers based on citing sentences[J]. Journal of data and information science, 2019, 4(4):26-41.
[29] AKSNES D W. Citation rates and perceptions of scientific contribution[J]. Journal of the American Society for Information Science and Technology, 2006, 57(2):169-185.
[30] 杨思洛,程濛,莫莹莹.学术论文研究亮点的语言特征与主题识别[J].图书馆论坛, 2023, 43(7):26-37.(YANG S L, CHENG M, MO Y Y. A multidimensional analysis of research highlights in academic papers:linguistic features and topic identification[J].Library tribune, 2023, 43(7):26-37.)
[31] HTOO T H H, NA JIN-CHEON. Comparison of Altmetrics across multiple disciplines:psychology, history, and linguistics.[C]//International conference of Asian special libraries 4th ICoASL. Seoul:The Special Libraries Association, 2015:1-12.
[32] GUMPENBERGER C, GLÄNZEL W, GORRAIZ J. The ecstasy and the agony of the altmetric score[J]. Scientometrics, 2016,108(2):977-982.
[33] OPENALEX. OpenAlex:end-to-end process for topic classification[EB/OL].[2025-04-16]. https://docs.google.com/document/d/1bDopkhuGieQ4F8gGNj7sEc8WSE8mvLZS/edit#heading=h.gi6h2p6vd793.
[34] DEGEN J, HAWKINS R D, GRAF C, et al. When redundancy is useful:a Bayesian approach to "overinformative" referring expressions[J]. Psychological review, 2020, 127(4):591.
[35] AYÁN C, FERNÁNDEZ-VILLA T, DURO A, et al. Reliability and validity of a questionnaire for assessing self-perceived health-related fitness in Spanish children[J]. The Spanish journal of psychology, 2020, 23:e25.
[36] OH V, TONG E M W. Negative emotion differentiation and longterm physical health-the moderating role of neuroticism[J].Health psychology, 2020, 39(2):127.
[37] PENG P, LIN X,ÜNAL Z E, et al. Examining the mutual relations between language and mathematics:a meta-analysis[J].Psychological bulletin, 2020, 146(7):595.
[38] PRICE G C, JANSEN K L, WEICK M R. Let's talk about sex:ethical considerations in survey research with minority populations[J]. Translational issues in psychological science,2020, 6(3):214.
[39] DEVLIN J, CHANG M W, LEE K, et al. Bert:pre-training of deep bidirectional transformers for language understanding[J].arXiv preprint, 2018, arXiv:1810.04805.
[40] 张蒙,王志灵,张昱城,等.元分析方法综述:从基础到高阶[J].技术经济, 2024, 43(1):140-151.(ZHANG M, WANG Z L,ZHANG Y C. A review of meta-analysis:from the basics to the advanced[J]. Journal of technology economics, 2024, 43(1):140-151.)
[41] DÍAZ-FAES A A, BOWMAN T D, COSTAS R. Towards a second generation of'social media metrics':characterizing Twitter communities of attention around science[J]. PloS one,2019, 14(5):e0216408.
[42] ZHANG L, GOU Z, FANG Z, et al. Who tweets scientific publications?a large-scale study of tweeting audiences in all areas of research[J]. Journal of the Association for Information Science and Technology, 2023, 74(13):1485-1497.
Outlines

/

〈 〉