SPECIAL TOPIC: Embracing the Challenges Presented by ChatGPT in Information Resource Management

Quality Evaluation and Application Strategies for ChatGPT-Generated Reviews

  • Lin Xin ,
  • Liu Zefei
Expand
  • School of Information Management, Central China Normal University, Wuhan 430079

Received date: 2023-11-02

  Revised date: 2024-02-16

  Online published: 2024-10-08

Supported by

This work is supported by the National Social Science Found of China titled “Research on Building a National Platform for High-End Communication of Scientific Research Papers and Scientific and Technological Information” (Grant No. 22ATQ006).

Abstract

[Purpose/Significance] This study aims to evaluate the quality of ChatGPT-generated research reviews and propose strategies for its application in scientific and technological intelligence work, providing a reference for promoting the application of ChatGPT in the field. [Method/Process] This study analyzed the characteristics of ChatGPT-generated research reviews and designed the quality evaluation criteria and methods. It selected three topics as samples to evaluate the quality of the generated reviews. Based on this, it proposed strategic suggestions for applying ChatGPT to scientific and technological intelligence work. [Results/Conclusion] ChatGPT-generated research reviews show potential value, with initial readability and some content reference. In terms of application advancement, on the one hand, the review generation capability of ChatGPT can be maximized to promote its direct application in scientific and technological intelligence work. On the other hand, considering the specific requirements of generating research reviews, secondary development and tool integration with ChatGPT can enhance its generation efficiency and effectiveness, thereby promoting its in-depth application.

Cite this article

Lin Xin , Liu Zefei . Quality Evaluation and Application Strategies for ChatGPT-Generated Reviews[J]. Library and Information Service, 2024 , 68(18) : 32 -40 . DOI: 10.13266/j.issn.0252-3116.2024.18.004

References

[1] 张智雄, 曾建勋, 夏翠娟, 等. 回应AIGC的信息资源管理学人思考[J]. 农业图书情报学报, 2023, 35(1): 4-28. (ZHANG Z X, ZENG J X, XIA C J, et al. Information resource management researchers' thinking about the opportunities and challenges of AIGC[J]. Journal of library and information science in agriculture, 2023, 35(1): 4-28.)
[2] BLANCO-GONZALEZ A, CABEZON A, SECO-GONZALEZ A, et al. The role of ai in drug discovery: challenges,opportunities, and strategies[J]. Pharmaceuticals, 2023, 16(6): 891-901.
[3] O'CONNOR S. Open artificial intelligence platforms in nursing education: tools for academic progress or abuse?[J]. Nurse education in practice, 2022, 66: 103537-103537.
[4] 陆伟, 刘家伟, 马永强, 等. ChatGPT为代表的大模型对信息资源管理的影响[J]. 图书情报知识, 2023, 40(2): 6-9, 70. (LU W, LIU J W, MA Y Q, et al. The influence of large language models represented by ChatGPT on information resources management[J]. Documentation, information & knowledge, 2023, 40(2): 6-9, 70.)
[5] 曹树金, 曹茹烨. 从ChatGPT看生成式AI对情报学研究与实践的影响[J]. 现代情报, 2023, 43(4): 3-10. (CAO S J, CAO R Y. Influence of generative Al on the research and practice of information science from the perspective of ChatGPT [J]. Journal of modern information, 2023, 43(4): 3-10.)
[6] 张智雄, 于改红, 刘熠, 等. ChatGPT对文献情报工作的影响[J]. 数据分析与知识发现, 2023, 7(3): 36-42. (ZHANG Z X, YU G H, LIU Y, et al. The influence of ChatGPT on library & information services[J]. Data analysis and knowledge discovery, 2023, 7(3): 36-42.)
[7] 刘琼, 刘桂锋, 卢章平, 等. 谋求与突破:ChatGPT对信息资源管理学科的影响与启示[J]. 信息资源管理学报, 2023, 13(5): 73-78. (LIU Q, LIU G F, LU Z P, et al. Seeking and breaking through: ChatGPT illuminating the way ahead for the discipline of information resource management[J]. Journal of information resources management, 2023, 13(5): 73-78.)
[8] 刘细文, 孙蒙鸽, 王茜, 等. DIKIW逻辑链下GPT大模型对文献情报工作的潜在影响分析[J]. 图书情报工作, 2023, 67(21): 3-12. (LIU X W, SUN M G, WANG Q, et al. Analysis of the potential impact of GPT large model under DIKIW logic chain on documentation and information services[J]. Library and information service, 2023, 67(21): 3-12.
[9] 郭亚军, 马慧芳, 张鑫迪, 等. ChatGPT赋能图书馆知识服务:原理、场景与进路[J/OL]. 图书馆建设: 1-16[2024-04-02]. http://kns.cnki.net/kcms/detail/23.1331.G2.20230713.1606.004.html. (GUO Y J, MA H F, ZHANG X D, et al. ChatGPT empowers library knowledge services: principles, scenarios, and approaches[J/OL]. Library development, 1-16.[2024-04-02]. http://kns.cnki.net/kcms/detail/23.1331.G2.20230713.1606.004.html.)
[10] 张强, 高颖, 赵逸淳, 等. ChatGPT在智慧图书馆建设中的机遇与挑战 [J]. 图书馆理论与实践, 2023, (6): 116-122. DOI:10.14064/j.cnki.issn1005-8214.20230815.001. (ZHANG Q, GAO Y, ZHAO Y C, et al. Opportunity and challenge of ChatGPT in the construction of intelligent library [J]. Library theory and practice, 2023, (6): 116-122. DOI:10.14064/j.cnki.issn1005-8214.20230815.001.)
[11] 郭亚军, 庞义伟, 周家华, 等. ChatGPT赋能图书馆虚拟数字人:技术优势、应用场景与实践路径[J/OL]. 图书馆论坛: 1-11. [2024-04-02]. http://kns.cnki.net/kcms/detail/44.1306.G2.20230706.0916.002.html. (GUO Y J, PANG Y W, ZHOU J H, et al. ChatGPT empowers digital humans in libraries: technological advantages application scenarios, and practical pathways[J/OL]. Library tribune, 1-11. [2024-04-02]. http://kns.cnki.net/kcms/detail/44.1306.G2.20230706.0916.002.html.)
[12] AITHAL S, AITHAL P S. Effects of AI-Based ChatGPT on higher education libraries[J]. International journal of management, technology, and social sciences, 2023, 8(2): 95-108.
[13] VERMA M. Novel study on AI-Based Chatbot (ChatGPT) impacts on the traditional library management[J]. International journal of trend in scientific research and development, 2023, 7(1): 961-964.
[14] HU Y, AMEER I, ZUO X, et al.Zero-shot clinical entity recognition using ChatGPT [J]. arXiv preprint arXiv:2303.16416.2023
[15] 鲍彤, 章成志.ChatGPT中文信息抽取能力测评——以三种典型的抽取任务为例[J]. 数据分析与知识发现, 2023, 7(9): 1-11. (BAO T, ZHANG C Z. Extracting Chinese information with ChatGPT: an empirical study by three typical tasks[J]. Data analysis and knowledge discovery, 2023, 7(9): 1-11.)
[16] RAHMAN M, TERANO HJR, RAHMAN N, et al. ChatGPT and academic research: a review and recommendations based on practical examples[J]. Journal of education, management and development studies, 2023, 3(1): 1-12.
[17] ALSHATER M. Exploring the role of artificial intelligence in enhancing academic performance: a case study of ChatGPT [EB/OL]. [2024-04-05]. https://papers.ssrn.ChatGPTom/sol3/papers.ChatGPTfm?abstraChatGPTt_id=4312358.
[18] ELSE H. Abstracts written by ChatGPT fool scientists[J]. Nature, 2023, 613(7944): 423.
[19] GAO C A, HOWARD F M, MARKOV N S, et al. Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers[J]. BioRxiv, 2022: 2022.12. 23.521610.
[20] 王一博, 郭鑫, 刘智锋, 等. AI生成与学者撰写中文论文摘要的检测与差异性比较研究[J]. 情报杂志, 2023, 42(9): 127-134. (WANG Y B, GUO X, LIU Z F, et al. Detection and comparative study of differences between Al-Generated and scholar-written Chinese abstracts[J]. Journal of intelligence, 2023, 42(9): 127-134.)
[21] KUTELA B, MSECHU K, DAS S, et al. Chatgpt's scientific writings: a case study on traffic safety[EB/OL]. [2024-04-05].https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4329120.
[22] 郭鑫, 王一博, 王继民. ChatGPT生成中文学术内容分析——以情报学领域为例[J/OL]. 图书馆论坛: 1-12. [2024-04-05]. http://kns.cnki.net/kcms/detail/44.1306.G2.20230921.1040.004.html. (GUO X, WANG Y B, WANG J M. Feature analysis of Chinese academic content generated by ChatGPT: taking the field of intelligence as an example[J/OL]. Library tribune, 1-12. [2024-04-05]. http://kns.cnki.net/kcms/detail/44.1306.G2.20230921.1040.004.html.)
[23] 白如江, 陈启明, 张玉洁, 等. 基于ChatGPT+Prompt的专利技术功效实体自动生成研究[J/OL]. 数据分析与知识发现: 1-15. [2024-04-05]. http://kns.cnki.net/kcms/detail/10.1478.G2.20231011.1120.006.html. (BAI R J, CHEN Q M, ZHANG Y J, et al. Generating effectiveness entities of patent technology based on ChatGPT+Prompt[J/OL]. Data analysis and knowledge discovery, 1-15. [2024-04-05]. http://kns.cnki.net/kcms/detail/10.1478.G2.20231011.1120.006.html.)
[24] CHEN J, CHEN P, WU X. Generating Chinese event extraction method based on ChatGPT and prompt learning[J]. Applied sciences, 2023, 13(17): 9500.
[25] 张恒, 赵毅, 章成志. 基于SciBERT与ChatGPT数据增强的研究流程段落识别[J/OL]. 情报理论与实践: 1-12. [2024-04-05]. http://kns.cnki.net/kcms/detail/11.1762.G3.20230816.1147.002.html. (ZHANG H, ZHAO Y, ZHANG C Z. Recognition of research workflow paragraphs based on SciBERT and ChatGPT data augmentation[J/OL]. Information studies:theory & application, 1-12. [2024-04-05]. http://kns.cnki.net/kcms/detail/11.1762.G3.20230816.1147.002.html.)
[26] AYDIN Ö, KARAARSLAN E. OpenAI ChatGPT generated literature review: digital twin in healthcare[J]. Emerging computer technologies, 2022, 2: 22-31.
[27] MA Y Q, LIU J W, YI F, et al. Al Vs.human—differentiation analysis of scientific content generation[EB/OL]. [2024-04-22]. https:/larxiw.orp!abs/2301.10416.
[28] 张智雄, 钱力, 谢靖, 等. Chat GPT对科学研究和文献情报工作的影响[R]. 北京:中国科学院文献情报中心, 国家科技文献图书中心, 2023. (ZHANG Z X, QIAN L, XIE J, et al. The Impact of ChatGPT on scientific research and bibliographic information work[R]. Beijing: National Science Library, Chinese Academy of Sciences, National Science and Technology Library, 2023.)
[29] 芬克. 如何做好文献综述[M]. 齐心, 译. 重庆: 重庆大学出版社, 2014. (FINK A. Conducting research literature reviews: from the Internet to paper[M]. QIN X, trans. Chongqing: Chongqing University Press, 2014.)
[30] ZHANG L, GHOSH R, DEKHIL M, et al. Combining lexicon-based and learning-based methods for Twitter sentiment analysis[J]. HP laboratories, technical report HPL-2011, 2011, 89: 1-8.
[31] ZHAO W X, ZHOU K, LI J, et al. A survey of large language models[J]. arXiv preprint arXiv:2303.18223, 2023.
[32] 车万翔, 窦志成, 冯岩松, 等. 大模型时代的自然语言处理:挑战、机遇与发展[J]. 中国科学:信息科学, 2023, 53(9): 1645-1687. (CHE W X, DOU Z C, FENG Y S, et al. Towards a comprehensive understanding of the impact of large language models on natural language processing: challenges, opportunities and future directions[J]. Scientia sinica (informationis), 2023, 53(9): 1645-1687.
Outlines

/