THEORETICAL STUDY

A Study on the Selection of Scientific Research Impact Evaluation Subjects Driven by the Development of Evaluation Theories

  • Wang Zuorong ,
  • Deng Sanhong ,
  • Wang Hao
Expand
  • 1 School of Information Management, Nanjing University, Nanjing 210023;
    2 Key Laboratory of Data Engineering and Knowledge Services in Provincial University (Nanjing University), Nanjing 210023

Received date: 2024-01-10

  Revised date: 2024-06-03

  Online published: 2024-11-25

Supported by

This work is supported by National Social Science Fund of China titled “Real-time Prediction and Long-term Value and Impact of Academic Achievements in the Big Data Environment”(Grant No.19BTQ062).

Abstract

[Purpose/Significance] The fourth-generation evaluation theory, with its core concept of collaborative construction, has driven the change of scientific research evaluation thinking. Under the background of this theory, the evaluation of the research impact also pays more attention to the process of impact generation and dissemination, and emphasizes the interactions of different stakeholders in the process. [Methodology/Process] Driven by both the categorized evaluation system at the policy level and the change of evaluation theory at the theoretical level, this study proposed the categorized evaluation idea of selecting evaluation subjects based on the content attributes of scientific knowledge. It further verified the rationality of this idea through real case data. This study took REF2021 case data as the main source of analysis, and employed prompt learning to assist manual identification of government, enterprise, university and public stakeholders. Based on it, it explored the differences in the participation of different stakeholder types in the research impact generation process in terms of the distribution of participation and cognitive themes. [Result/Conclusion] The results demonstrate differences in the subjective cognition and realistic participation levels of the four stakeholder types in the research impact generation process. On the basis of the confirmed rationality of evaluation ideas, this study proposes suggestions for the selection of evaluation subjects based on disciplinary category from the specific case data, and provides new practical ideas for academic evaluation practices in China.

Cite this article

Wang Zuorong , Deng Sanhong , Wang Hao . A Study on the Selection of Scientific Research Impact Evaluation Subjects Driven by the Development of Evaluation Theories[J]. Library and Information Service, 2024 , 68(22) : 45 -57 . DOI: 10.13266/j.issn.0252-3116.2024.22.005

References

[1] 国务院办公厅.国务院办公厅关于完善科技成果评价机制的指导意见[EB/OL].[2024-05-01].https://www.gov.cn/gongbao/content/2021/content_5631817.htm.(The General Office of the State Council.Guiding Opinions of the General Office of the State Council on Improving the Evaluation Mechanism of Scientific and Technological Achievements[EB/OL].[2024-05-01].https://www.gov.cn/gongbao/content/2021/content_5631817.htm.)
[2] 李正风, 阎妍, 武晨箫.学术评价破“五唯”与科学共同体的责任[J].科学学与科学技术管理, 2022, 43(10): 17-28.(LI Z F, YAN Y, WU C X.Breaking the “Five-only” in academic evaluation and the responsibility of the science community[J].Science of science and management of S.&.T, 2022, 43(10): 17-28.)
[3] 苏新宁, 王东波.学术评价相关问题与思考[J].信息资源管理学报, 2018, 8(3): 4-11.(SU X N, WANG D B.Related questions and thoughts of academic evaluation[J].Journal of information resources management, 2018, 8(3): 4-11.)
[4] 邓三鸿, 杨杰, 王昊, 等.多源异构数据视角下的学术评价:内涵、进展与展望[J].科技情报研究, 2023, 5(4): 42-56.(DENG S H, YANG J, WANG H, et al.Academic evaluation from the perspective of multi-source heterogeneous data:connotation, status and prospect [J].Scientific information research, 2023, 5(4): 42-56.)
[5] 沈小玲, 徐勇, 严卫中.基于网络文献计量的科技论文学术影响力综合评价研究[J].图书情报工作, 2013, 57(21): 95-103.(SHEN X L, XU Y, YAN W Z.Empirical research on network comprehensive evaluation of academic influence of sci-tech papers[J].Library and information service, 2013, 57(21): 95-103.)
[6] 孙志茹, 张志强.基于知识运用理论的学术研究政策影响力评价框架研究[J].图书情报工作, 2013, 57(8): 40-45.(SUN Z R, ZHANG Z Q.Evaluation framework for policy influence of academic research based on knowledge utilization theory[J].Library and information service, 2013, 57(8): 40-45.)
[7] BOZEMAN B, YOUTIE J.Socio-economic impacts and public value of government-funded research: lessons from four US national science foundation initiatives[J].Research policy, 2017, 46(8): 1387-1398.
[8] SALTER A J, MARTIN B R.The economic benefits of publicly funded basic research: a critical review[J].Research policy, 2001, 30(3): 509-532.
[9] 曾燕, 昝婷婷, 杨晓, 等.科技成果的文化影响概念框架与REF2021案例研究[J].数据分析与知识发现, 2024, 8(5): 139-150.(ZENG Y, ZAN T T, YANG X, et al.Conceptual framework for cultural impacts of scientific research and REF2021 cases[J].Data analysis and knowledge discovery, 2024, 8(5): 139-150.)
[10] 姜春林, 郭琪琴, 张光耀.人文社科学术著作评价指标体系构建及实证研究[J].情报杂志, 2022, 41(2): 198-207.(JIANG C L, GUO Q Q, ZHANG G Y.Construction and empirical study of evaluation indicators system for chinese academic books in humanities and social sciences[J].Journal of intelligence, 2022, 41(2): 198-207.)
[11] 韩松涛, 李洁, 金佳丽, 等.基于关联重言的人文社科期刊评价量化模型探索[J].情报学报, 2023, 42(6): 702-710.(HAN S T, LI J, JIN J L, et al.Quantitative evaluation model for humanities and social sciences journals based on association tautology [J].Journal of the China Society for Scientific and Technical Information, 2023, 42(6): 702-710.)
[12] 袁曦临, 刘宇.人文社会科学评价的复杂性与引文评价指标的修正[J].图书情报工作, 2010, 54(14): 24-27, 96.(YUAN X L, LIU Y.Comment on the essence and complexity of humanity social sciences evaluation[J].Library and information service, 2010, 54(14): 24-27, 96.)
[13] 马海群, 韩娜, 张春春.社会科学应用成果评价体系构建研究[J].情报资料工作, 2022, 43(4): 44-51.(MA H Q, HAN N, ZHANG C C.A probe into the evaluation system construction of social sciences application achievements[J].Information and documentation services, 2022, 43(4): 44-51.)
[14] 郭红梅, 何苗, 邢星, 等.不同自然科学门类间论文学术影响力多指标综合评价的合理性研究[J].图书情报工作, 2012, 56(22): 62-68.(GUO H M, HE M, XING X, et al.The construction of the comprehensive evaluation system and its feasible study on the academic influence of papers across natural scientific disciplines[J].Library and information service, 2012, 56(22): 62-68.)
[15] 顾东蕾.生物信息学学术期刊评价[J].中华医学图书情报杂志, 2009, 18(4): 72-76.(GU D L.Evaluation of academic bioinformatics journals[J].Chinese journal of medical library and information science, 2009, 18(4): 72-76.)
[16] 王颖婕, 柳卸林, 王雪璐, 等.科研项目学术价值评价及影响因素研究[J].科学学研究, 2020, 38(3): 409-417.(WANG Y J, LIU X L, WANG X L, et al.Gender disparities in academic commercialization and its influencing factors[J].Studies in science of science, 2020, 38(3): 409-417.)
[17] PRAUS P.HCR for assessment of scientific journals in chemistry[J].Scientometrics, 2020, 122(2): 1237-1242.
[18] 杨冠灿, 刘彤, 李纲, 等.基于综合引用网络的专利价值评价研究[J].情报学报, 2013, 32(12): 1265-1277.(YANG G C, LIU T, LI G, et al.Research on patent value evaluation based on comprehensive citation network[J].Journal of the China Society for Scientific and Technical Information, 2013, 32(12): 1265-1277.)
[19] 伊惠芳, 吴红, 马永新, 等.改进的柔性专利价值评价方法——基于时域-领域双重视角[J].情报杂志, 2019, 38(3): 53-60.(YI H F, WU H, MA Y X, et al.Improved flexible patent value evaluation method——based on the dual perspective of time domain and technology field[J].Journal of intelligence, 2019, 38(3): 53-60.)
[20] 康旭东, 邓乐乐, 王宇开, 等.基于全代引证的专利累积影响力评价——一个诺奖得主专利的案例研究[J].情报学报, 2021, 40(3): 267-277.(KANG X L, DENG L L, WANG Y K, et al.Evaluation of patents cumulative impact based on all generations of citations: a case study of a nobel prize winner’s patents[J].Journal of the China Society for Scientific and Technical Information, 2021, 40(3): 267-277.)
[21] 孙明汉, 江曼, 朱秀珠.基于影响域和活跃度的专利即时影响力评价——以基因工程领域为例[J].情报理论与实践, 2023, 46(4): 159-166.(SUN M H, JIANG M, ZHU X Z.Patent current impact index based on domain of influence and activity: a case study in genetic engineering[J].Information studies: theory & application, 2023, 46(4): 159-166.)
[22] 郭凤娇, 赵蓉英, 孙劭敏.基于科学交流过程的学术论文影响力评价研究——以中国社会科学国际学术论文为例[J].情报学报, 2020, 39(4): 357-366.(GUO F J, ZHAO R Y, SUN S M.Evaluation of academic papers impact based on scientific communication path: a case study of chinese international academic papers in social sciences [J].Journal of the China Society for Scientific and Technical Information, 2020, 39(4): 357-366.)
[23] 牌艳欣, 周彦廷.生态位理论视角下学术论文影响力评价研究[J].情报理论与实践, 2022, 45(12): 119-127, 145.(PAI Y X, ZHOU Y T.Research on impact evaluation of academic papers from the perspective of niche theory[J].Information studies: theory & application, 2022, 45(12): 119-127, 145.)
[24] 许鑫, 叶丁菱.多维影响力融合视域下的数据论文评价研究[J].情报学报, 2022, 41(3): 275-286.(XU X, YE D L.Evaluation of data papers from the perspective of multi-dimensional influence fusion[J].Journal of the China Society for Scientific and Technical Information, 2022, 41(3): 275-286.)
[25] 张玉, 潘云涛, 袁军鹏, 等.论多维视角下中文科技图书学术影响力评价体系的构建[J].图书情报工作, 2015, 59(7): 69-76.(ZHANG Y, PAN Y T, YUAN J P, et al.The construction of academic impact evaluation system of sci-tech books in china under multiple dimensions[J].Library and information service, 2015, 59(7): 69-76.)
[26] 章成志, 童甜甜, 周清清.整合不同评论平台的图书综合影响力评价研究[J].情报学报, 2018, 37(9): 861-873.(ZHANG C Z, TONG T T, ZHOU Q Q.Combing multiple platforms of online reviews to measure the comprehensive impact of books[J].Journal of the China Society for Scientific and Technical Information, 2018, 37(9): 861-873.)
[27] 熊霞.基于PlumX的外文学术图书影响力评价实证研究[J].四川图书馆学报, 2021(3): 34-40.(XIONG X.Empirical research on the evaluation of foreign academic book impact based on PlumX[J].Journal of the library science society of Sichuan, 2021(3): 34-40.)
[28] 王萝娜, 李端明.运用加权TOPSIS法的学术图书影响力评价方法研究[J].图书情报工作, 2018, 62(23): 72-77.(WANG L N, LI D M.Research on the evaluation method of academic book influence using weighted TOPSIS method[J].Library and information service, 2018, 62(23): 72-77.)
[29] 朱富成, 刘永.利用政策关联度指标进行学术影响力研究——以疫情防控档案政策关联度为例[J].图书情报工作, 2023, 67(14): 62-72.(ZHU F C, LIU Y.Discussion on academic influence of analysis of policy related degree index of our country: a case study of the correlation degree of epidemic prevention and control archives policies[J].Library and information service, 2023, 67(14): 62-72.)
[30] 翟姗姗, 叶丁菱, 胡畔, 等.融合Altmetrics与引文分析的数据论文学术影响力评价[J].情报学报, 2020, 39(7): 710-718.(ZHAI S S, YE D L, HU P, et al.Evaluation of the academic impact of data papers fused with altmetrics and citation analysis[J].Journal of the China Society for Scientific and Technical Information, 2020, 39(7): 710-718.)
[31] 王菲菲, 弋新月, 贾晨冉, 等.Altmetrics视角下科技文献学术影响力动态评价体系构建与实证研究[J].情报理论与实践, 2020, 43(8): 77-83.(WANG F F, YI X Y, JIA C R, et al.Construction and empirical research on dynamic evaluation system of academic influence of scientific literature from the perspective of altmetrics[J].Information studies: theory & application, 2020, 43(8): 77-83.)
[32] 何春建.单篇论文学术影响力评价指标构建[J].图书情报工作, 2017, 61(4): 98-107.(HE C J.Study on the evaluation indicators of single article academic impact[J].Library and information service, 2017, 61(4): 98-107.)
[33] 曾粤亮, 郑汉, 杨思洛.人文科学研究成果的社会影响探析——以REF案例为样本[J].情报学报, 2023, 42(7): 842-856.(ZENG Y L, ZHENG H, YANG S L.Societal impact indicators of research outcomes in humanities based on REF case studies[J].Journal of the China Society for Scientific and Technical Information, 2023, 42(7): 842-856.)
[34] 陈芳, 杨建林.科研成果社会影响力评价研究——评价方法视角下的解读[J].现代情报, 2021, 41(11): 111-119.(CHEN F, YANG J L.The societal impact evaluation on scientific research achievements: interpretation from the perspective of evaluation method[J].Journal of modern information, 2021, 41(11): 111-119.)
[35] GUBA E G, LINCOLN Y S.Fourth generation evaluation[M].New York: Sage Publications, Inc, 1989.
[36] 古贝, 林肯.第四代评估[M].秦林, 蒋爱玲, 译.北京: 中国人民大学出版社, 2008: 24-30.(GUBA E G, LINCOLN Y S.Fourth generation evaluation[M].QIN L, JIANG A L, trans.Beijing: Renmin University of China Press, 2008: 24-30.)
[37] 张民选.回应、协商与共同建构——“第四代评价理论”评述[J].外国教育资料, 1995(3): 53-59.(ZHANG M X.Response, negotiation and co-construction: a review of the fourth generation of evaluation theory[J].Global Education, 1995(3): 53-59.)
[38] 温萍.论“第四代评价理论”对我国本科教学评估的启示[J].中国成人教育, 2010(17): 135-136.(WEN P.The implications of the fourth generation evaluation theory for undergraduate teaching evaluation in China[J].China adult education, 2010(17): 135-136.)
[39] 刘佳.第四代评价理论视阈下高校教学评价制度的反思与重建[J].教育发展研究, 2015, 35(17): 56-61.(LIU J.On higher education assessment system in the perspective of the fourth assessment theory[J].Research in educational development, 2015, 35(17): 56-61.)
[40] 方向阳.基于第四代评价理论的高职院校创业实践评价[J].教育与职业, 2015(14): 74-75.(FANG X Y.Evaluation of entrepreneurial practices in higher education institutions based on fourth generation evaluation theory[J].Education and vocation, 2015(14): 74-75.)
[41] 孙科技, 朱益明.“双一流”建设评估的现实困境及其超越:第四代评估理论视角[J].复旦教育论坛, 2021, 19(4): 100-106.(SUN K J, ZHU Y M.The realistic dilemma and transcendence of “double first-class” construction evaluation: from the perspective of the fourth generation evaluation theory[J].Fudan education forum, 2021, 19(4): 100-106.)
[42] 杨彩菊.第四代评价理论下高职学生学习评价反思[J].职业技术教育, 2015, 36(13): 39-43.(YANG C J.Reflection on student learning evaluation of higher vocational education in the view of the fourth generation evaluation theory[J].Vocational and technical education, 2015, 36(13): 39-43.)
[43] 周香均, 罗志敏.关于研究生综合素质测评的思考——基于第四代评价理论[J].教育探索, 2016(3): 95-99.(ZHOU X J, LUO Z M.Reflections on comprehensive quality assessment of graduate students: based on the fourth generation evaluation theory[J].Education exploration, 2016(3): 95-99.)
[44] 中华人民共和国教育部.国家中长期教育改革和发展规划纲要(2010-2020年)[EB/OL].[2024-01-02].http://www.moe.gov.cn/jyb_xwfb/s6052/moe_838/201008/t20100802_93704.html.(Ministry of Education of the People’s Republic of China.Outline of the National Medium- and Long-Term Education Reform and Development Plan (2010-2020) [EB/OL].[2024-01-02].http://www.moe.gov.cn/jyb_xwfb/s6052/moe_838/201008/t20100802_93704.html.)
[45] 陈秋媛, 李想.第四代教育评价理论视角下的博士研究生学术创新成果综合评价——以北京大学为例[J].北京教育(高教), 2022(4): 18-20.(CHEN Q Y, LI X.Fourth generation educational evaluation theory perspective comprehensive evaluation of academic innovation achievements of doctoral students——taking Peking University as an example[J].Beijing education, 2022(4): 18-20.)
[46] 郭瑞, 杨天通.高校智库评价指标体系的构建及实证研究——基于第四代评估理论视角[J].智库理论与实践, 2021, 6(5): 33-44.(GUO R, YANG T T.Construction and empirical research of evaluation index system of university think tank: based on the perspective of the fourth generation evaluation theory[J].Think tank: theory & practice, 2021, 6(5): 33-44.)
[47] 文雯, 李雪, 王晶.第四代评估理论视角下的研究生项目评估[J].高等工程教育研究, 2015(3): 108-113.(WEN W, LI X, WANG J.Program evaluation of postgraduates from the perspective of the fourth-generation evaluation theory[J].Research in higher education of engineering, 2015(3): 108-113.)
[48] PEARSE H.The educational imagination[J].Canadian journal of education, 1980, 5(3): 112-115.
[49] HUEBNER A J.A constructivist approach to evaluation methodology: implications for positive youth development[D].Tucson: The University of Arizona, 1995.
[50] 蔡亮华, 陈伟俊, 杨红梅, 等.商业计划书投资人眼里的好方案[M].北京: 企业管理出版社, 2022: 189.(CAI L H, CHEN W J, YANG H M, et al.Business plan: a good program in the eyes of investors [M].Beijing: Enterprise Management Press, 2022: 189.)
[51] TANTALO C, PRIEM R L.Value creation through stakeholder synergy[J].Strategic management journal, 2016, 37(2): 314-329.
[52] AMIRI S, KING D R, DUESING R J.Managing divestments as projects: benefits of stakeholder orientation[J].International journal of project management, 2022, 40(4): 385-397.
[53] GARRIGA E.Beyond stakeholder utility function: stakeholder capability in the value creation process[J].Journal of business ethics, 2014, 120(4): 489-507.
[54] LONG K, WANG Y, ZHAO Y, et al.Who are the stakeholders and how do they respond to a local government payments for ecosystem services program in a developed area: a case study from Suzhou, China[J].Habitat international, 2015, 49: 1-9.
[55] BUCHHOLZ R A, ROSENTHAL S B.Stakeholder theory and public policy: how governments matter[J].Journal of business ethics, 2004, 51(2): 143-153.
[56] ROSE J, FLAK L S, SAEBO O.Stakeholder theory for the e-government context: framing a value-oriented normative core[J].Government information quarterly, 2018, 35(3): 362-374.
[57] 俞立平, 潘云涛, 武夷山.基于利益相关者分析的学术期刊评价指标选择研究[J].中国科技论坛, 2009(8): 20-23.(YU L P, PAN Y T, WU Y S.Research on the selection of evaluation indicators for academic journals based on stakeholder analysis[J].Forum on science and technology in China, 2009(8): 20-23.)
[58] 姜帅, 贝政新.利益相关者视角下科技期刊可持续发展研究[J].编辑学报, 2019, 31(2): 141-144.(JIANG S, BEI Z X.Sustainable development of scientific journals from stakeholder perspective[J].Acta editologica, 2019, 31(2): 141-144.)
[59] 谭春辉.高校人文社会科学研究成果评价机理研究——基于利益相关者的视角[J].社会科学管理与评论, 2013(2): 16-23, 111.(TAN C H.Study on evaluation mechanism of university’s humanities and social sciences research achievement: based on the perspective of stakeholders[J].Management and review of social sciences, 2013(2): 16-23, 111.)
[60] 尤莉.大学学术资源指标优先性评价及配置路径——基于对不同利益相关者的实证调查[J].现代教育管理, 2017(10): 43-49.(YOU L.Priority evaluation and configuration path of university academic resources[J].Modern education management, 2017(10): 43-49.)
[61] 刘黎明.从主体缺位到共同治理: 中美高校科研评价比较研究[J].科技管理研究, 2022, 42(3): 50-56.(LIU L M.From subject’s absence to co-governance: a comparative study of research evaluation systems in Chinese and American universities[J].Science and technology management research, 2022, 42(3): 50-56.)
[62] REF.About REF[EB/OL].[2023-12-24].https://2021.ref.ac.uk/about-the-ref/index.html.
[63] ARC.Excellence in research for Australia[EB/OL].[2024-02-23].https://www.arc.gov.au/evaluating-research/excellence-research-australia#excellence-in-research-for-australia.
[64] ARC.Past ERA evaluations[EB/OL].[2024-01-08].https://www.arc.gov.au/evaluating-research/excellence-research-australia/past-era-evaluation.
[65] 刘晓娟, 周若卿.国外科研项目绩效评价实践及启示[J].图书情报工作, 2023, 67(14): 119-129.(LIU X J, ZHOU R Q.Practice and enlightenment of performance evaluation of scientific research projects abroad[J].Library and information service, 2023, 67(14): 119-129.)
[66] KONG J, ZHANG L X, JIANG M, et al.Incorporating multi-level CNN and attention mechanism for Chinese clinical named entity recognition[J].Journal of biomedical informatics, 2021, 116: 103737.
[67] MAO Y R, HAO Y, LIU W W, et al.Class-imbalanced-aware distantly supervised named entity recognition[EB/OL].[2024-08-18].https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10109246.
[68] ALSAARAN N, ALRABIAH M.Classical arabic named entity recognition using variant deep neural network architectures and BERT[J].IEEE access, 2021, 9: 91537-91547.
[69] YE F Y, HUANG L, LIANG S J, et al.Decomposed two-stage prompt learning for few-shot named entity recognition[J].Information, 2023, 14(5): 262.
[70] ZHAO B, JIN W Q, DEL SER J, et al.ChatAgri: exploring potentials of ChatGPT on cross-linguistic agricultural text classification[J].Neurocomputing, 2023, 557: 126708.
[71] DATTA S, LEE K, PAEK H, et al.AutoCriteria: a generalizable clinical trial eligibility criteria extraction system powered by large language models[J].Journal of the American Medical Informatics Association, 2024, 31(2): 375-385.
[72] WOOLLEY R, MOLAS-GALLART J.Research impact seen from the user side[J].Research evaluation, 2023, 32(3): 591-602.
[73] COHEN J.A power primer[J].Psychological bulletin, 1992, 112(1): 115-159.
[74] Cambridge University Press Assessment.Cambridge dictionary[EB/OL].[2024-02-23].https://dictionary.cambridge.org/us/dictionary/english/.
Outlines

/

〈 〉