SPECIAL TOPIC: Scientific Innovation and Knowledge Transformation from the Perspective of New Quality Productive Forces

Policy Impact Evaluation of Academic Papers Using Text Association Analysis: A Case Study in Artificial Intelligence

  • Jin Jieqin ,
  • Yan Yuwei
Expand
  • 1 Nanjing Tech University Library, Nanjing 210009;
    2 School of Economics and Management, Nanjing Tech University, Nanjing 211816

Received date: 2025-05-16

  Revised date: 2025-08-06

  Online published: 2025-12-05

Abstract

[Purpose/Significance] Academic papers are the main form of disseminating and validating scientific knowledge. Integrating such knowledge into policymaking can improve the scientificity and rationality of documents. By evaluating the policy impact, it can effectively demonstrate how academic papers can be transformed into policy and explore a new perspective on the evaluation of academic paper impact. [Method/Process] This paper adopts text mining methods, such as Sentence-BERT, SBERTopic to construct an indicator system for evaluating policy impact of academic papers based on semantic association from three dimensions: mention-level, thematic potential, and sentence-level impact. An empirical analysis is conducted on academic papers and policy texts in artificial intelligence from 2010 to 2024. The entropy weight-TOPSIS method is applied to weight the indicators for a composite policy impact score. [Result/Conclusion] Only a few papers reveal high policy impact, with their topic distribution and content exhibiting a clear tendency. Under different evaluation criteria, there are significant differences in the influence of papers in the field of artificial intelligence. In the future, scholars and policymakers should collaborate to address the challenges in translating academic research into policy, establish and improve the platform of government think tanks, promote more and faster academic papers into the policy-making process, and explore a more comprehensive and effective evaluation system.

Cite this article

Jin Jieqin , Yan Yuwei . Policy Impact Evaluation of Academic Papers Using Text Association Analysis: A Case Study in Artificial Intelligence[J]. Library and Information Service, 2025 , 69(24) : 31 -43 . DOI: 10.13266/j.issn.0252-3116.2025.24.003

References

[1] 国务院办公厅. 国务院办公厅关于完善科技成果评价机制的指导意见[EB/OL]. [2025-10-01]. https://www.mee.gov.cn/zcwj/gwywj/202108/t20210802_853757.shtml. (General Office of the State Council. Guiding opinions of the general office of the state council on improving the evaluation mechanism for scientific and technological achievements[EB/OL]. [2025-10-01]. https://www.mee.gov.cn/zcwj/gwywj/202108/t20210802_853757.shtml.)
[2] 董鲁皖龙. 破“SCI至上”后, 科研评价体系如何立[N]. 中国教育报, 2020-03-18(4). (DONG LUWANLONG. After breaking the “SCI supremacy”, how to establish a scientific research evaluation system[N]. China education daily, 2020-03-18(4).)
[3] Cheng X, Tang L, ZHOU M, et al. Coevolution of COVID-19 research and China’s policies[J]. Health research policy and systems, 2021, 19(1): 121.
[4] 任超, 杨孟辉, 李恺, 等. 新冠肺炎疫情防控政策中引用科学论文的影响因素分析[J]. 情报学报, 2023, 42(3): 341-353. (REN C, YANG M H, LI K, et al. Analysis of factors in citing scientific papers in policies against COVID-19 pandemic[J]. Journal of the China Society for Scientific and Technical Information, 2023, 42(3): 341-353.)
[5] 杨代福, 沈玲丽. 政策制定对学术研究的使用:程度测量、影响因素、对策及启示[J]. 智库理论与实践, 2024, 9(2): 13-21, 28. (YANG D F, SHEN L L. The use of academic research in policy making: degree measurement, influencing factors, countermeasures and enlightenments [J]. Think tank: theory & practice, 2024, 9(2): 13-21, 28.)
[6] 王楠, 罗珺文. 英国高校科研成果的社会影响力评析——基于REF2014评估中6679份影响力案例的再分析[J]. 清华大学教育研究, 2019, 40(5): 75-81. (WANG N, LUO J W. The impact of scientific research from UK higher education institutions based on a synthetic analysis of the 6679 impact case studies in ref2014[J]. Tsinghua journal of education, 2019, 40(5): 75-81.)
[7] 赵蓉英, 郭凤娇, 谭洁. 基于Altmetrics的学术论文影响力评价研究——以汉语言文学学科为例[J]. 中国图书馆学报, 2016, 42(1): 96-108. (ZHAO R Y, GUO F J, TAN J. Evaluation of academic papers impact based on Altmetrics: a case study of Chinese language and literature[J]. Journal of library science in China, 2016, 42(1): 96-108.)
[8] 张义川, 赵聪聪, 伍军红, 等. 期刊政策影响力评价方法研究[J]. 图书情报工作, 2022, 66(22): 95-105. (Zhang Y C, Zhao C C, Wu J H, et al. Research on evaluation method of journal policy influence[J]. Library and information service, 2022, 66(22): 95-105.)
[9] Weiss C H. The many meanings of research utilization[J]. Public administration review, 1979, 39(5): 426-431.
[10] Lavis J N, Ross S E, Hurley J E, et al. Examining the role of health services research in public policymaking[J]. The milbank quarterly, 2002, 80(1): 125-154.
[11] Bornmann L, Haunschild R, Marx W. Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?[J]. Scientometrics, 2016, 109(3): 1477-1495.
[12] 曹喆, 尚媛媛, 张琳. 哪些研究更易产生政策影响——以学术论文分类与多维特征分析为路径[J]. 图书情报知识, 2025, 42(1): 6-17, 28. (CAO Z, SHANG Y Y, ZHANG L. What kinds of research are more likely to generate policy impact: taking classification and multidimensional characteristics of academic articles as pathways[J]. Documentation, information & knowledge, 2025, 42(1): 6-17, 28.)
[13] 余厚强, 肖婷婷, 王曰芬, 等. 政策文件替代计量指标分布特征研究[J]. 中国图书馆学报, 2017, 43(5): 57-69. (YU H q, XIAO T t, WANG Y f, et al. Study of distribution characteristics of policy documents Altmetrics [J]. Journal of library science in China, 2017, 43(5): 57-69.)
[14] Dorta-González P, Rodríguez-Caro A, Dorta-González M I. Societal and scientific impact of policy research: a large-scale empirical study of some explanatory factors using Altmetric and Overton[J]. Journal of informetrics, 2024, 18(3): 101530.
[15] Newson R, Rychetnik L, King L, et al. Does citation matter? Research citation in policy documents as an indicator of research impact: an Australian obesity policy case-study[J]. Health research policy and systems, 2018, 16(1): 55.
[16] 刘晓娟, 周若卿, 肖云彤, 等. 政策提及指标在学术成果社会影响力评价中的应用价值分析[J]. 情报理论与实践, 2024, 47(11): 47-55. (Liu X J, Zhou R Q, Xiao Y T, et al. Application value analysis of policy mention indicators in the evaluation of social impact of academic achievements[J]. Information studies: theory & application, 2024, 47(11): 47-55.)
[17] Yu H, Murat B, Li J, et al. How can policy document mentions to scholarly papers be interpreted? An analysis of the underlying mentioning process[J]. Scientometrics, 2023, 128(11): 6247-6266.
[18] 朱富成, 刘永. 利用政策关联度指标进行学术影响力研究——以疫情防控档案政策关联度为例[J]. 图书情报工作, 2023, 67(14): 62-72. (Zhu F C, Liu Y. Discussion on academic influence of analysis of policy related degree index of our country: a case study of the correlation degree of epidemic prevention and control archives policies[J]. Library and information service, 2023, 67(14): 62-72.)
[19] 胡吉明, 钱玮, 文鹏, 等. 基于结构功能和实体识别的文本语义表示——以病历领域为例[J]. 数据分析与知识发现, 2022, 6(8): 110-121. (Hu J M, Qian W, Wen P, et al. Text semantic representation with structure-function and entity recognition: case study of medical records[J]. Data analysis and knowledge discovery, 2022, 6(8): 110-121.)
[20] 冉从敬, 田文芳, 贾志轩. 基于混合方法的“科学论文-专利技术”关联关系模型构建——以生物医药领域为例[J]. 情报科学, 2024, 42(6): 132-143. (RAN C J, TIAN W f, JIA Z x. Modeling of scientific paper-patent technology association relationship based on mixed methods: taking the biomedical field as an example[J]. Information science, 2024, 42(6): 132-143.)
[21] 武楷彪, 郎宇翔, 董瑜. 融合句法结构和词义信息的政策文本关联挖掘方法研究[J]. 数据分析与知识发现, 2022, 6(5): 20-33. (Wu K B, Lang Y X, Dong Y. Mining policy text relevance with syntactic structure and semantic information[J]. Data analysis and knowledge discovery, 2022, 6(5): 20-33.)
[22] Riley B L, Kernoghan A, Stockton L, et al. Using contribution analysis to evaluate the impacts of research on policy: getting to ‘good enough’[J]. Research evaluation, 2018, 27(1): 16-27.
[23] Tattersall A, Carroll C. What can altmetric.com tell us about policy citations of research? An analysis of altmetric.com data for research articles from the university of Sheffield[J]. Frontiers in research metrics and analytics, 2017, 2(1): 9.
[24] 李萍. 我国上市公司年报中“管理层讨论与分析”的可读性分析[J]. 时代金融, 2018(9): 225-226. (Li P. Readability analysis of “management discussion and analysis” in the annual reports of listed companies in China[J]. Times finance, 2018(9): 225-226.)
[25] Ren C, Yang M. Study on the characteristics of cross-domain knowledge diffusion from science to policy: evidence from Overton data[J]. Proceedings of the Association for Information Science and Technology, 2023, 60(1): 368-378.
[26] 刘自强, 许海云, 罗瑞, 等. 基于主题关联分析的科技互动模式识别方法研究[J]. 情报学报, 2019, 38(10): 997-1011. (Liu Z q, Xu H y, Luo R, et al. Research on scientific and technological interaction patterns based on topic relevance analysis[J]. Journal of the China Society For Scientific And Technical Information, 2019, 38(10): 997-1011.)
[27] 阮光册, 周萌葳. 基于Sentence-BERT的专利技术主题聚类研究——以人工智能领域为例[J]. 情报杂志, 2024, 43(2): 110-117. (Ruan G c, Zhou M w. Research on patent technology subject clustering based on Sentence-BERT: taking the field of artificial intelligence as an example[J]. Journal of intelligence, 2024, 43(2): 110-117.)
[28] 唐晓波, 吴海婷, 吴佳琳. 基于特征知识元的专利语义引用识别方法研究——以量子计算领域为例[J]. 情报理论与实践, 2023, 46(10): 86-95. (TANG X B, WU H T, WU J L. Research on patent semantic references recognition method based on feature knowledge elements: using quantum computing as an example[J]. Information studies: theory & application, 2023, 46(10): 86-95.)
[29] Inui K, Jiang J, Ng V, et al. Sentence-BERT: sentence embeddings using Siamese Bert-networks[C]// Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing. Hong Kong: Association for Computational Linguistics, 2019: 3982-3992.
[30] Bornmann L, Haunschild R, Boyack K, et al. How relevant is climate change research for climate change policy? An empirical analysis based on Overton data[J]. Plos one, 2022, 17(9): e0274693.
Outlines

/

〈 〉