图书情报工作

图书情报工作 >

2025 , Vol. 69 >Issue 10: 84 - 96

DOI: https://doi.org/10.13266/j.issn.0252-3116.2025.10.008

研究论文

大语言模型生成与学者撰写论文摘要对比分析与识别——以情报学领域为例

  • 王伟正 ,
  • 乔鸿 ,
  • 李肖俊 ,
  • 王静静
展开
  • 1 山东师范大学图书馆 济南 250358;
    2 山东师范大学商学院 济南 250358;
    3 齐鲁工业大学(山东省科学院)数字人文研究中心 济南 250014;
    4 齐鲁工业大学(山东省科学院)情报研究所 济南 250014;
    5 山东大学新闻传播学院 济南 250100
王伟正，硕士研究生；乔鸿，副教授，博士，硕士生导师，通信作者，E-mail：qiaohongsd@126.com；李肖俊，研究员，博士；王静静，副研究员，博士，硕士生导师。

收稿日期: 2024-07-12

  修回日期: 2024-11-11

  网络出版日期: 2025-05-16

基金资助

本文系国家自然科学基金青年项目“基于多源异构数据的科技关键节点及信息扩散机理研究”（项目编号：72304169）研究成果之一。

收起

Comparative Analysis and Identification Between Abstracts Generated by Large Language Models and Written by Scholars: A Case Study in the Field of Information Science

  • Wang Weizheng ,
  • Qiao Hong ,
  • Li Xiaojun ,
  • Wang Jingjing
Expand
  • 1 Shandong Normal University Library, Jinan 250358;
    2 Shandong Normal University Business School, Jinan 250358;
    3 Digital Humanities Research Center, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250014;
    4 Institute of Information Science, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250014;
    5 School of Journalism and Communication, Shandong University, Jinan 250100
Wang Weizheng, master candidate; Qiao Hong, associate professor, PhD, master supervisor, corresponding author, E-mail: qiaohongsd@126.com; Li Xiaojun, researcher, PhD; Wang Jingjing, associate researcher, PhD, master supervisor.

Received date: 2024-07-12

  Revised date: 2024-11-11

  Online published: 2025-05-16

Supported by

This work is supported by the youth project of the National Natural Science Foundation of China, titled “Research on the Mechanism of Key Scientific and Technological Nodes and Information Diffusion Based on Multi-source Heterogeneous Data” (Grant No. 72304169).

Fold

摘要

[目的/意义] 探索大语言模型生成与学者撰写论文摘要的差异性，为学术论文的AIGC检测提供参考。[方法/过程] 以情报学领域近三年高被引论文为例，首先使用Prompt根据论文标题生成对应摘要，构建研究数据集；其次，从图灵测试、词汇特征、词性特征、困惑度、主题一致性评测对两类文本的异同进行深入分析，揭示两类文本的差异性；最后，提出一种基于BERT-CNN的分类器对两类摘要文本进行识别。[结果/结论] 当前人工无法较好识别大语言模型生成的论文摘要，其识别的精确率甚至低于随机猜测概率0.5；大语言模型生成的摘要相对较长，使用的词汇量也较多，两类摘要在名词占比上具有较大差异；学者撰写论文摘要拥有更高的困惑度，两类摘要的部分主题分布一致，在关注焦点、研究视角上存在较大差异；BERT-CNN的分类器具有最好的分类效果，超过主流的五种机器学习模型和三种深度学习模型。

关键词： 大语言模型; 文本特征; 文本分类; 论文摘要; ChatGPT

本文引用格式

王伟正 , 乔鸿 , 李肖俊 , 王静静 . 大语言模型生成与学者撰写论文摘要对比分析与识别——以情报学领域为例[J]. 图书情报工作, 2025 , 69(10) : 84 -96 . DOI: 10.13266/j.issn.0252-3116.2025.10.008

Abstract

[Purpose/Significance] Exploring the differences between abstracts generated by large language models and those written by scholars can provide references for AIGC detection in academic papers. [Method/Process] Taking highly cited papers in the field of information science in the past three years as examples, this paper used prompts to generate corresponding abstracts based on the paper titles and construct a research dataset. Subsequently, it conducted an in-depth analysis of the similarities and differences between the two types of texts through Turing tests, lexical features, part-of-speech features, perplexity, and thematic consistency, to reveal their differences. Finally, a BERT-CNN-based classifier was proposed to identify the two types of abstract texts. [Result/Conclusion] Currently, humans are unable to effectively identify abstracts generated by large language models, with an accuracy rate even lower than the random guessing probability of 0.5. Abstracts generated by large language models tend to be relatively longer and use a larger vocabulary, with significant differences in the proportion of nouns between the two types of abstracts. Abstracts written by scholars have higher perplexity, and while some thematic distributions of the two types of abstracts are consistent, there are significant differences in focus and research perspectives. The BERT-CNN classifier exhibits the best classification performance, surpassing five mainstream machine learning models and three deep learning models.

Key words： large language model; textual features; text classification; academic abstract; ChatGPT

参考文献

[1] PELAU C, DABIJIA D C, ENE I. What makes an AI device human-like? The role of interaction quality, empathy and perceived psychological anthropomorphic characteristics in the acceptance of artificial intelligence in the service industry[J]. Computers in human behavior, 2021(122): 106855.
[2] RAY P P. ChatGPT: a comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope[J]. Internet of things and cyber-physical systems, 2023(3): 121-154.
[3] TAECHARUNGROJ V. “What can ChatGPT do?” analyzing early reactions to the innovative AI chatbot on Twitter[J]. Big data and cognitive computing, 2023, 7(1): 35.
[4] The Oxford Scientist. ChatGPT: a milestone in natural language processing[EB/OL]. [2025-04-12]. https://oxsci.org/chatgpt-natural-language-processing/.
[5] VESELOVSKY V, RIBEIRO M H, WEST R. Artificial intelligence: crowd workers widely use large language models for text production tasks[EB/OL]. [2025-04-12]. https://doi.org/10.48550/arXiv.2306.07899.
[6] COTTON D R E, COTTON P A, SHIPWAY J R. Chatting and cheating: ensuring academic integrity in the era of ChatGPT[J]. Innovations in education and teaching international, 2024, 61(2): 228-239.
[7] ADESHOLA I, ADEPOJU A P. The opportunities and challenges of ChatGPT in education[J]. Interactive learning environments, 2023: 1-14.
[8] O’CONNOR S. Open artificial intelligence platforms in nursing education: tools for academic progress or abuse?[J]. Nurse education in practice, 2022, 66: 103537-103537.
[9] 魏顺平, 范学健, 王向旭, 等.高等教育应用ChatGPT的潜能与风险——来自美国高校的经验与启示[J/OL]. 现代远距离教育:1-15[2025-04-12]. https://doi.org/10.13927/j.cnki.yuan.20240628.003. (WEI S P, FAN X J, WANG X X, et al. The potential and risks of applying ChatGPT in higher education: experience and inspiration of American universities[J/OL]. Modern distance education:1-15[2025-04-12]. https://doi.org/10.13927/j.cnki.yuan.20240628.003.)
[10] HOLDEN O L, NORRIS M E, KUHLMEIER V A. Academic integrity in online assessment: a research review[J]. Frontiers in education, 2021(6): 639814.
[11] ELALI F R, RACHID L N. AI-generated research paper fabrication and plagiarism in the scientific community[J]. Patterns, 2023, 4(3): 100706.
[12] 中华人民共和国中央人民政府. 中华人民共和国学位法[EB/OL]. [2025-04-12]. https://www.gov.cn/yaowen/liebiao/202404/content_6947841.htm. (The Central People’s Government of the People’s Republic of China. Degree law of the people’s republic of China[EB/OL]. [2025-04-12]. https://www.gov.cn/yaowen/liebiao/202404/content_6947841.htm.)
[13] 图书情报工作. Al政策声明[EB/OL]. [2025-04-12]. https://www.lis.ac.cn/CN/column/column27.shtml. (Library and information service.AI policy statement[EB/OL]. [2025-04-12]. https://www.lis.ac.cn/CN/column/column27.shtml.)
[14] OpenAl. ChatGPT: optimizing language models for dialogue[EB/OL]. [2025-04-12]. https://openai.com/chatgpt.
[15] 陆伟, 刘家伟, 马永强, 等. ChatGPT为代表的大模型对信息资源管理的影响[J]. 图书情报知识, 2023, 40(2): 6-9. (LU W, LIU J W, MA Y Q, et al. The influence of large language models represented by ChatGPT on information resources management[J]. Documentation, informaiton & knowledge, 2023, 40(2): 6-9.)
[16] 曹树金, 曹茹烨. 从ChatGPT看生成式AI对情报学研究与实践的影响[J]. 现代情报, 2023, 43(4): 3-10. (CAO S J, CAO R Y. Influence of generative AI on the research and practice of information science from the perspective of ChatGPT. Journal of modern information, 2023, 43(4): 3-10)
[17] 王静静, 叶鹰, 王婉茹. GPT类技术应用开启智能信息处理之颠覆性变革[J]. 图书馆杂志, 2023, 42(5): 9-13. (WANG J J, YE Y, WANG W R. Subversive change for intelligent information processing by the application of GPT-Type technology[J]. Library journal, 2023, 42(5): 9-13.)
[18] DWIVEDI Y K, KSHETRI N, HUGHES L, et al. Opinion paper: “so what if ChatGPT wrote it?” multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy[J]. International journal of information management, 2023(71): 102642.
[19] 刘细文, 孙蒙鸽, 王茜, 等. DIKIW逻辑链下GPT大模型对文献情报工作的潜在影响分析[J]. 图书情报工作, 2023, 67(21): 3-12. (LIU X W, SUN M G, WANG X, et al. Analysis of the potential impact of GPT large model under DIKIW logic chain on documentation and information services[J]. Library and information service, 2023, 67(21): 3-12.)
[20] LUND B D, WANG T, MANNURU N R, et al. ChatGPT and a new academic reality: artificial intelligence‐written research papers and the ethics of the large language models in scholarly publishing[J]. Journal of the Association for Information Science and Technology, 2023, 74(5): 570-581.
[21] 聂思言杨江华. 多维视角下新一代人工智能技术的公众感知研究[J]. 情报杂志, 2024, 43(9): 130-138. (NIE S Y, YANG J H. Public perception of new generation artificial intelligence technology from a multidimensional perspective[J]. Journal of intelligence, 2024, 43(9): 130-138.)
[22] 王伟正, 乔鸿, 李肖俊, 等. 基于AIDUA框架的生成式人工智能使用意愿研究[J]. 农业图书情报学报, 2024, 36(2): 36-50. (WANG W Z, QIAO H, LI X J, et al. User willingness to use generative artificial intelligence based on AIDUA framework[J]. Journal of library and information science in agriculture, 2024, 36(2): 36-50.)
[23] ELSE H. Abstracts written by ChatGPT fool scientists[J]. Nature, 2023, 613(7944): 423
[24] ELKHATAT A M, ELSAID K, ALMEER S. Evaluating the efficacy of AI content detection tools in differentiating between human and AI-generated text[J]. International journal for educational integrity, 2023, 19(1): 17.
[25] KUTELA B, MSECHU K, DAS S, et al. Chatgpt’s scientific writings: a case study on traffic safety[DB/OL]. [2025-04-12]. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4329120.
[26] GAO C A, HOWARD F M, MARKOV N S, et al. Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers[J]. NPJ digital medicine, 2023, 6(1): 75.
[27] 施亦龙, 许鑫. ChatGPT机器回答与知乎人工回答的比较[J]. 图书馆论坛, 2024, 44(1): 151-159. (SHI Y L, XU X. A Comparative study on the response characteristics of ChatGPT and Zhihu[J]. Library tribune, 2024, 44(1): 151-159.)
[28] 王一博, 郭鑫, 刘智锋, 等. AI生成与学者撰写中文论文摘要的检测与差异性比较研究[J]. 情报杂志, 2023, 42(9): 127-134. (WANG Y B, GUO X, LIU Z F, et al. Detection and comparative study of differences between AI-generated and scholar-written Chinese abstracts[J]. Journal of intelligence, 2023, 42(9): 127-134.)
[29] 张强, 王潇冉, 高颖, 等. ChatGPT生成与学者撰写文献摘要的对比研究——以信息资源管理领域为例[J]. 图书情报工作, 2024, 68(8): 35-47. (ZHANG Q, WANG X R, GAO Y, et al. Comparative study on ChatGPT and scholars’ abstracts: taking the field of information resource management as an example[J]. Library and information service, 2024, 68(8): 35-47.)
[30] 林鑫, 刘泽妃. ChatGPT生成综述的质量评测与应用策略[J]. 图书情报工作, 2024, 68(18): 32-40. (LIN X, LIU Z F. Quality evaluation and application strategies for ChatGPT-generated reviews[J]. Library and information service, 2024, 68(18): 32-40.)
[31] NIGH M. ChatGPT3 prompt engineering[EB/OL]. [2025-04-12]. https://github.com/mattnigh/ChatGPT3-Free-Prompt-List.
[32] TURING A M. Computing machinery and intelligence[M]. Netherlands: Springer, 2009.
[33] CUI Y, CHE W, LIU T, et al. Pre-training with whole word masking for Chinese Bert[J]. IEEE/ACM transactions on audio, speech, and language processing, 2021(29): 3504-3514.
[34] HUGGING FACE. Wenzhong-GPT2-110M[DB/OL]. [2025-04-12]. https://huggingface.co/IDEA-CCNL/Wenzhong-GPT2-110M.
[35] CHENG S L, TSAI S J, BAI Y M, et al. Comparisons of quality, correctness, and similarity between ChatGPT-generated and human-written abstracts for basic research: cross-sectional study[J]. Journal of medical Internet research, 2023(25): e51229.
Options
文章导航

/

〈 〉