图书情报工作

图书情报工作 >

2024 , Vol. 68 >Issue 8: 35 - 47

DOI: https://doi.org/10.13266/j.issn.0252-3116.2024.08.004

工作研究

ChatGPT生成与学者撰写文献摘要的对比研究——以信息资源管理领域为例

  • 张强 ,
  • 王潇冉 ,
  • 高颖 ,
  • 王常珏 ,
  • 周洪
展开
  • 1 安徽工程大学人文学院 芜湖 241000;
    2 华中师范大学信息管理学院 武汉 430079;
    3 安徽工程大学计算机与信息学院 芜湖 241000;
    4 中国科学院大学经济与管理学院 北京 101190;
    5 中国科学院武汉文献情报中心 武汉 430071
张强,博士研究生;王潇冉,硕士研究生;高颖,硕士研究生;周洪,副研究员,博士研究生。

收稿日期: 2023-09-05

  修回日期: 2023-12-31

  网络出版日期: 2024-05-17

基金资助

本文系安徽省省级教学研究重大项目“新工科背景下人文素质教育体系的改革与实践”(项目编号:2020jyxm0152)和中国科学院武汉文献情报中心青年领军人才项目(项目编号:E0KZ451)研究成果之一。

收起

Comparative Study on ChatGPT and Scholars' Abstracts: Taking the Field of Information Resource Management as an Example

  • Zhang Qiang ,
  • Wang Xiaoran ,
  • Gao Ying ,
  • Wang Chang jue ,
  • Zhou Hong
Expand
  • 1 School of Humanities, Anhui Polytechnic University, Wuhu 241000;
    2 School of Information Management, Central China Normal University, Wuhan 430079;
    3 School of Computer and Information, Anhui Polytechnic University, Wuhu 241000;
    4 Department of Information Resources Management, School of Economics and Management, University of Chinese Academy of Sciences, Beijing 100191;
    5 Wuhan Library and Intelligence Center of Chinese Academy of Science, Wuhan 430071

Received date: 2023-09-05

  Revised date: 2023-12-31

  Online published: 2024-05-17

Supported by

This work is supported by Anhui Provincial teaching and Research project "Reform and Practice of Humanistic Quality Education System under the Background of New Engineering"(Grant No.2020jyxm0152) and Wuhan Literature and Information Center of Chinese Academy of Sciences Young Leading Talents Project (Grant No.E0KZ451)

Fold

摘要

[目的/意义] 探究ChatGPT生成与学者撰写的中文论文摘要之间的异同，为AI生成学术论文检测及相关研究提供借鉴。[方法/过程] 首先，以信息资源管理领域为例，分别抽取图书馆学、情报学、档案学近3年各500篇高被引论文，基于获取的论文题目采用Prompt方式应用ChatGPT工具生成对应的摘要文本，构建数据集合；其次，采用9种机器学习及深度学习算法对ChatGPT生成与学者撰写的摘要文本进行分类检测；最后，从文本特征、主题模型、ROUGE评测对二者的异同进行多角度分析，从而揭示二者之间的异同点。[结果/结论] 基于数据集所训练的主流机器学习及深度学习算法可以有效地分辨摘要是AI生成还是学者撰写，其中BERT和ERNIE的效果最好，而机器学习算法中RF和Xgboost效果最好。ChatGPT生成的摘要字符数量、句子数量较学者撰写的要多，

关键词： ChatGPT; 文本分类; 文本特征; 论文摘要

本文引用格式

张强 , 王潇冉 , 高颖 , 王常珏 , 周洪 . ChatGPT生成与学者撰写文献摘要的对比研究——以信息资源管理领域为例[J]. 图书情报工作, 2024 , 68(8) : 35 -47 . DOI: 10.13266/j.issn.0252-3116.2024.08.004

Abstract

[Purpose/Significance] To explore the similarities and differences of Chinese abstracts written by ChatGPT and scholars can provide references for AI-generated academic paper detection and related research.[Method/Process] Firstly,taking the field of information resource management as an example,this paper extracted 500 highly cited papers from library science,information science,and archival science in the recent years.Based on the obtained paper titles,it used the prompt method to apply the ChatGPT tool to generate corresponding abstract texts and construct a dataset.Secondly,it adopted 9 machine learning and deep learning algorithms to classify and detect abstract texts generated by ChatGPT and written by scholars.Finally,it analyzed and revealed the similarities and differences between the two from multiple perspectives,including text features,topic models,and ROUGE evaluation.[Result/Conclusion] Mainstream machine learning and deep learning algorithms trained on datasets can effectively distinguish whether abstracts are generated by AI or written by scholars,with BERT and ERNIE performing best,while RF and Xgboost best in machine learning algorithms.The number of abstract characters and sentences generated by ChatGPT is more than that written by scholars,and the

Key words： ChatGPT; text classification; text features; paper abstract

参考文献

[1] WANG F Y, LI J, QIN R, et al. ChatGPT for computational social systems:from conversational applications to human-oriented operating systems[J]. IEEE transactions on computational social systems, 2023, 10(2):414-425.
[2] MONDAL S, DAS S, VRANA V G. How to bell the cat?A theoretical teview of generative artificial intelligence towards digital disruption in all walks of life[J]. Technologies, 2023, 11(2):44.
[3] WU T, HE S, LIU J, et al. A brief overview of ChatGPT:The history, status quo and potential future development[J]. IEEE/CAA Journal of Automatica Sinica, 2023, 10(5):1122-1136.
[4] ADESHOLA I, ADEPOJU A P. The opportunities and challenges of ChatGPT in education[J/OL]. Interactive learning environments, 2023:1-14. https://doi.org/10.1080/10494820.2023.2253858.
[5] Nature. Initial submission[EB/OL].[2024-03-06]. https://www.nature.com/nature/for-authors/initial-submission.
[6] 图书情报工作. AI政策声明[EB/OL].[2024-03-06]. https://www.lis.ac.cn/CN/column/column27.shtml.(Library and information service. AI policy statement[EB/OL].[2024-03-06]. https://www.lis.ac.cn/CN/column/column27.shtml.)
[7] STOKRL-WALKER C, VAN NOORDEN R. What ChatGPT and generative AI mean for science[J]. Nature, 2023, 614(7947):214-216.
[8] BUDHWAR P, CHOWDHURY S, WOOD G, et al. Human resource management in the age of generative artificial intelligence:perspectives and research directions on ChatGPT[J]. Human resource management journal, 2023, 33(3):606-659.
[9] 戴岭,胡姣,祝智庭. ChatGPT赋能教育数字化转型的新方略[J].开放教育研究, 2023, 29(4):41-48.(DAI L, HU J, ZHU Z T. ChatGPT empowering the new strategy of digital transformation in education[J]. Open education research, 2023, 29(4):41-48.)
[10] 陆伟,刘家伟,马永强,等. ChatGPT为代表的大模型对信息资源管理的影响[J].图书情报知识, 2023, 40(2):6-9,70.(LU W, LIU J W, MA Z Q, et al. The impact of large models represented by ChatGPT on information resource management[J]. Documentation, information&knowledge, 2023, 40(2):6-9, 70.)
[11] 张智雄,于改红,刘熠,等. ChatGPT对文献情报工作的影响[J].数据分析与知识发现, 2023, 7(3):36-42.(ZHANG Z Q, YU G H, LIU Y, et al. The impact of ChatGPT on document information work[J]. Data analysis and knowledge discovery, 2023, 7(3):36-42.)
[12] LUND B D, WANG T. Chatting about ChatGPT:how may AI and GPT impact academia and libraries?[J]. Library hi tech news, 2023, 40(3):26-29.
[13] 曹树金,曹茹烨.从ChatGPT看生成式AI对情报学研究与实践的影响[J].现代情报, 2023, 43(4):3-10.(CAO S J, CAO R Y. Exploring the impact of generative AI on research and practice in information science[J]. Modern information, 2023, 43(4):3-10.)
[14] 周文欢. ChatGPT在档案领域应用和意义[J].中国档案, 2023(3):62-63.(ZHOU W H. Application and significance of ChatGPT in the field of archives[J]. China archives, 2023(3):62-63.)
[15] ZHENG H, ZHAN H. ChatGPT in scientific writing:a cautionary tale[J]. The American journal of medicine, 2023, 136(8):725-726.
[16] WALTERS W H, WILDER E I. Fabrication and errors in the bibliographic citations generated by ChatGPT[J]. Scientific Reports, 2023, 13(1):14045.
[17] 张华平,李林翰,李春锦. ChatGPT中文性能测评与风险应对[J].数据分析与知识发现, 2023, 7(3):16-25.(ZHANG H P, LI L H, LI C J. Performance evaluation and risk response of ChatGPT in Chinese[J]. Data analysis and knowledge discovery, 2023, 7(3):16-25.)
[18] 鲍彤,章成志. ChatGPT中文信息抽取能力测评——以三种典型的抽取任务为例[J].数据分析与知识发现, 2023, 7(9):1-11.(BAO T, ZHANG C Z. Evaluation of ChatGPT's Chinese information extraction ability:taking three typical extraction tasks as examples[J]. Data analysis and knowledge Discovery, 2023, 7(9):1-11.)
[19] 施亦龙,许鑫. ChatGPT机器回答与知乎人工回答的比较[J].图书馆论坛, 2024, 44(1):151-159.(SHI Y L, XU X. A comparison between ChatGPT machine answers and Zhihu artificial answers[J]. Library tribune, 2024, 44(1):151-159.)
[20] NIGH M. ChatGPT3 prompt engineering[EB/OL].[2024-03-06]. https://github.com/mattnigh/ChatGPT3-Free-Prompt-List.
[21] BLEI D M, NG A Y, JORDAN M I. Latent dirichlet allocation[J]. Journal of machine learning research, 2003, 3:993-1022.
[22] LIN C Y. ROUGE:A Package for Automatic Evaluation of Summaries[C]//Text Summarization Branches Out. Barcelona, Spain:Association for Computational Linguistics, 2004:74-81.
[23] DEVLIN J, CHANG M W, LEE K, et al. BERT:Pre-training of Deep Bidirectional Transformers for Language Understanding[C]//BURSTEIN J, DORAN C, SOLORIO T. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies, Volume 1(Long and Short Papers). Minneapolis, Minnesota:Association for Computational Linguistics, 2019:4171-4186.
[24] SUN Y, WANG S, LI Y, et al. Ernie:enhanced representation through knowledge integration[EB/OL].[2024-03-06]. https://arxiv.org/abs/1904.09223.
[25] 王红斌,金子铃,毛存礼.结合层级注意力的抽取式新闻文本自动摘要[J].计算机科学与探索, 2022, 16(4):877-887.(WANG H B, JIN Z L, MAO C L. Hierarchical attention-based extractive news text summarization[J]. Journal of frontiers of computer science and technology, 2022, 16(4):877-887.)
[26] 赵江江,王洋,许楹楹,等.基于知识蒸馏的抽取式自动摘要模型[J].计算机科学, 2023, 50(S1):214-220.(ZHAO J J, WANG Y, XU Y Y, et al. Extractive summarization model based on knowledge distillation[J]. Computer science, 2023, 50(S1):214-220.)
Options
文章导航

/

〈 〉