专题:迎接ChatGPT对信息资源管理的挑战

ChatGPT生成综述的质量评测与应用策略

  • 林鑫 ,
  • 刘泽妃
展开
  • 华中师范大学信息管理学院 武汉 430079
林鑫,副教授,博士,硕士生导师,论文选题与框架确定,论文撰写与修改,E-mail:xinlin@ccnu.edu.cn;刘泽妃,硕士研究生,进行实验,论文撰写与修改

收稿日期: 2023-11-02

  修回日期: 2024-02-16

  网络出版日期: 2024-10-08

基金资助

本文系国家社会科学基金重点项目“国家科研论文和科技信息高端交流平台构建研究”(项目编号:22ATQ006)研究成果之一。

Quality Evaluation and Application Strategies for ChatGPT-Generated Reviews

  • Lin Xin ,
  • Liu Zefei
Expand
  • School of Information Management, Central China Normal University, Wuhan 430079

Received date: 2023-11-02

  Revised date: 2024-02-16

  Online published: 2024-10-08

Supported by

This work is supported by the National Social Science Found of China titled “Research on Building a National Platform for High-End Communication of Scientific Research Papers and Scientific and Technological Information” (Grant No. 22ATQ006).

摘要

[目的/意义] 对ChatGPT生成研究综述的质量进行评价,并提出其应用策略,为推进ChatGPT在科技情报工作方面的应用提供参考。[方法/过程] 在分析ChatGPT生成研究综述特点的基础上,设计ChatGPT生成综述的评价指标与方法,并选择3个主题作为样本对其生成的综述质量进行了评测,在此基础上提出将其应用于科技情报工作的策略建议。[结果/结论] ChatGPT生成的研究综述在内容上具有一定的参考价值并初步达到了可读性要求,具备一定的应用价值。应用推进中,一方面,可以尽量挖掘综述生成能力推进其在科技情报工作中的直接应用;另一方面,也可以结合研究综述生成这一特定场景要求,进行ChatGPT的二次开发与工具集成,在提升其生成的效率与效果基础上推进深度应用。

本文引用格式

林鑫 , 刘泽妃 . ChatGPT生成综述的质量评测与应用策略[J]. 图书情报工作, 2024 , 68(18) : 32 -40 . DOI: 10.13266/j.issn.0252-3116.2024.18.004

Abstract

[Purpose/Significance] This study aims to evaluate the quality of ChatGPT-generated research reviews and propose strategies for its application in scientific and technological intelligence work, providing a reference for promoting the application of ChatGPT in the field. [Method/Process] This study analyzed the characteristics of ChatGPT-generated research reviews and designed the quality evaluation criteria and methods. It selected three topics as samples to evaluate the quality of the generated reviews. Based on this, it proposed strategic suggestions for applying ChatGPT to scientific and technological intelligence work. [Results/Conclusion] ChatGPT-generated research reviews show potential value, with initial readability and some content reference. In terms of application advancement, on the one hand, the review generation capability of ChatGPT can be maximized to promote its direct application in scientific and technological intelligence work. On the other hand, considering the specific requirements of generating research reviews, secondary development and tool integration with ChatGPT can enhance its generation efficiency and effectiveness, thereby promoting its in-depth application.

参考文献

[1] 张智雄, 曾建勋, 夏翠娟, 等. 回应AIGC的信息资源管理学人思考[J]. 农业图书情报学报, 2023, 35(1): 4-28. (ZHANG Z X, ZENG J X, XIA C J, et al. Information resource management researchers' thinking about the opportunities and challenges of AIGC[J]. Journal of library and information science in agriculture, 2023, 35(1): 4-28.)
[2] BLANCO-GONZALEZ A, CABEZON A, SECO-GONZALEZ A, et al. The role of ai in drug discovery: challenges,opportunities, and strategies[J]. Pharmaceuticals, 2023, 16(6): 891-901.
[3] O'CONNOR S. Open artificial intelligence platforms in nursing education: tools for academic progress or abuse?[J]. Nurse education in practice, 2022, 66: 103537-103537.
[4] 陆伟, 刘家伟, 马永强, 等. ChatGPT为代表的大模型对信息资源管理的影响[J]. 图书情报知识, 2023, 40(2): 6-9, 70. (LU W, LIU J W, MA Y Q, et al. The influence of large language models represented by ChatGPT on information resources management[J]. Documentation, information & knowledge, 2023, 40(2): 6-9, 70.)
[5] 曹树金, 曹茹烨. 从ChatGPT看生成式AI对情报学研究与实践的影响[J]. 现代情报, 2023, 43(4): 3-10. (CAO S J, CAO R Y. Influence of generative Al on the research and practice of information science from the perspective of ChatGPT [J]. Journal of modern information, 2023, 43(4): 3-10.)
[6] 张智雄, 于改红, 刘熠, 等. ChatGPT对文献情报工作的影响[J]. 数据分析与知识发现, 2023, 7(3): 36-42. (ZHANG Z X, YU G H, LIU Y, et al. The influence of ChatGPT on library & information services[J]. Data analysis and knowledge discovery, 2023, 7(3): 36-42.)
[7] 刘琼, 刘桂锋, 卢章平, 等. 谋求与突破:ChatGPT对信息资源管理学科的影响与启示[J]. 信息资源管理学报, 2023, 13(5): 73-78. (LIU Q, LIU G F, LU Z P, et al. Seeking and breaking through: ChatGPT illuminating the way ahead for the discipline of information resource management[J]. Journal of information resources management, 2023, 13(5): 73-78.)
[8] 刘细文, 孙蒙鸽, 王茜, 等. DIKIW逻辑链下GPT大模型对文献情报工作的潜在影响分析[J]. 图书情报工作, 2023, 67(21): 3-12. (LIU X W, SUN M G, WANG Q, et al. Analysis of the potential impact of GPT large model under DIKIW logic chain on documentation and information services[J]. Library and information service, 2023, 67(21): 3-12.
[9] 郭亚军, 马慧芳, 张鑫迪, 等. ChatGPT赋能图书馆知识服务:原理、场景与进路[J/OL]. 图书馆建设: 1-16[2024-04-02]. http://kns.cnki.net/kcms/detail/23.1331.G2.20230713.1606.004.html. (GUO Y J, MA H F, ZHANG X D, et al. ChatGPT empowers library knowledge services: principles, scenarios, and approaches[J/OL]. Library development, 1-16.[2024-04-02]. http://kns.cnki.net/kcms/detail/23.1331.G2.20230713.1606.004.html.)
[10] 张强, 高颖, 赵逸淳, 等. ChatGPT在智慧图书馆建设中的机遇与挑战 [J]. 图书馆理论与实践, 2023, (6): 116-122. DOI:10.14064/j.cnki.issn1005-8214.20230815.001. (ZHANG Q, GAO Y, ZHAO Y C, et al. Opportunity and challenge of ChatGPT in the construction of intelligent library [J]. Library theory and practice, 2023, (6): 116-122. DOI:10.14064/j.cnki.issn1005-8214.20230815.001.)
[11] 郭亚军, 庞义伟, 周家华, 等. ChatGPT赋能图书馆虚拟数字人:技术优势、应用场景与实践路径[J/OL]. 图书馆论坛: 1-11. [2024-04-02]. http://kns.cnki.net/kcms/detail/44.1306.G2.20230706.0916.002.html. (GUO Y J, PANG Y W, ZHOU J H, et al. ChatGPT empowers digital humans in libraries: technological advantages application scenarios, and practical pathways[J/OL]. Library tribune, 1-11. [2024-04-02]. http://kns.cnki.net/kcms/detail/44.1306.G2.20230706.0916.002.html.)
[12] AITHAL S, AITHAL P S. Effects of AI-Based ChatGPT on higher education libraries[J]. International journal of management, technology, and social sciences, 2023, 8(2): 95-108.
[13] VERMA M. Novel study on AI-Based Chatbot (ChatGPT) impacts on the traditional library management[J]. International journal of trend in scientific research and development, 2023, 7(1): 961-964.
[14] HU Y, AMEER I, ZUO X, et al.Zero-shot clinical entity recognition using ChatGPT [J]. arXiv preprint arXiv:2303.16416.2023
[15] 鲍彤, 章成志.ChatGPT中文信息抽取能力测评——以三种典型的抽取任务为例[J]. 数据分析与知识发现, 2023, 7(9): 1-11. (BAO T, ZHANG C Z. Extracting Chinese information with ChatGPT: an empirical study by three typical tasks[J]. Data analysis and knowledge discovery, 2023, 7(9): 1-11.)
[16] RAHMAN M, TERANO HJR, RAHMAN N, et al. ChatGPT and academic research: a review and recommendations based on practical examples[J]. Journal of education, management and development studies, 2023, 3(1): 1-12.
[17] ALSHATER M. Exploring the role of artificial intelligence in enhancing academic performance: a case study of ChatGPT [EB/OL]. [2024-04-05]. https://papers.ssrn.ChatGPTom/sol3/papers.ChatGPTfm?abstraChatGPTt_id=4312358.
[18] ELSE H. Abstracts written by ChatGPT fool scientists[J]. Nature, 2023, 613(7944): 423.
[19] GAO C A, HOWARD F M, MARKOV N S, et al. Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers[J]. BioRxiv, 2022: 2022.12. 23.521610.
[20] 王一博, 郭鑫, 刘智锋, 等. AI生成与学者撰写中文论文摘要的检测与差异性比较研究[J]. 情报杂志, 2023, 42(9): 127-134. (WANG Y B, GUO X, LIU Z F, et al. Detection and comparative study of differences between Al-Generated and scholar-written Chinese abstracts[J]. Journal of intelligence, 2023, 42(9): 127-134.)
[21] KUTELA B, MSECHU K, DAS S, et al. Chatgpt's scientific writings: a case study on traffic safety[EB/OL]. [2024-04-05].https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4329120.
[22] 郭鑫, 王一博, 王继民. ChatGPT生成中文学术内容分析——以情报学领域为例[J/OL]. 图书馆论坛: 1-12. [2024-04-05]. http://kns.cnki.net/kcms/detail/44.1306.G2.20230921.1040.004.html. (GUO X, WANG Y B, WANG J M. Feature analysis of Chinese academic content generated by ChatGPT: taking the field of intelligence as an example[J/OL]. Library tribune, 1-12. [2024-04-05]. http://kns.cnki.net/kcms/detail/44.1306.G2.20230921.1040.004.html.)
[23] 白如江, 陈启明, 张玉洁, 等. 基于ChatGPT+Prompt的专利技术功效实体自动生成研究[J/OL]. 数据分析与知识发现: 1-15. [2024-04-05]. http://kns.cnki.net/kcms/detail/10.1478.G2.20231011.1120.006.html. (BAI R J, CHEN Q M, ZHANG Y J, et al. Generating effectiveness entities of patent technology based on ChatGPT+Prompt[J/OL]. Data analysis and knowledge discovery, 1-15. [2024-04-05]. http://kns.cnki.net/kcms/detail/10.1478.G2.20231011.1120.006.html.)
[24] CHEN J, CHEN P, WU X. Generating Chinese event extraction method based on ChatGPT and prompt learning[J]. Applied sciences, 2023, 13(17): 9500.
[25] 张恒, 赵毅, 章成志. 基于SciBERT与ChatGPT数据增强的研究流程段落识别[J/OL]. 情报理论与实践: 1-12. [2024-04-05]. http://kns.cnki.net/kcms/detail/11.1762.G3.20230816.1147.002.html. (ZHANG H, ZHAO Y, ZHANG C Z. Recognition of research workflow paragraphs based on SciBERT and ChatGPT data augmentation[J/OL]. Information studies:theory & application, 1-12. [2024-04-05]. http://kns.cnki.net/kcms/detail/11.1762.G3.20230816.1147.002.html.)
[26] AYDIN Ö, KARAARSLAN E. OpenAI ChatGPT generated literature review: digital twin in healthcare[J]. Emerging computer technologies, 2022, 2: 22-31.
[27] MA Y Q, LIU J W, YI F, et al. Al Vs.human—differentiation analysis of scientific content generation[EB/OL]. [2024-04-22]. https:/larxiw.orp!abs/2301.10416.
[28] 张智雄, 钱力, 谢靖, 等. Chat GPT对科学研究和文献情报工作的影响[R]. 北京:中国科学院文献情报中心, 国家科技文献图书中心, 2023. (ZHANG Z X, QIAN L, XIE J, et al. The Impact of ChatGPT on scientific research and bibliographic information work[R]. Beijing: National Science Library, Chinese Academy of Sciences, National Science and Technology Library, 2023.)
[29] 芬克. 如何做好文献综述[M]. 齐心, 译. 重庆: 重庆大学出版社, 2014. (FINK A. Conducting research literature reviews: from the Internet to paper[M]. QIN X, trans. Chongqing: Chongqing University Press, 2014.)
[30] ZHANG L, GHOSH R, DEKHIL M, et al. Combining lexicon-based and learning-based methods for Twitter sentiment analysis[J]. HP laboratories, technical report HPL-2011, 2011, 89: 1-8.
[31] ZHAO W X, ZHOU K, LI J, et al. A survey of large language models[J]. arXiv preprint arXiv:2303.18223, 2023.
[32] 车万翔, 窦志成, 冯岩松, 等. 大模型时代的自然语言处理:挑战、机遇与发展[J]. 中国科学:信息科学, 2023, 53(9): 1645-1687. (CHE W X, DOU Z C, FENG Y S, et al. Towards a comprehensive understanding of the impact of large language models on natural language processing: challenges, opportunities and future directions[J]. Scientia sinica (informationis), 2023, 53(9): 1645-1687.
文章导航

/