Library and Information Service

Library and Information Service >

2024 , Vol. 68 >Issue 8: 35 - 47

DOI: https://doi.org/10.13266/j.issn.0252-3116.2024.08.004

SERVICE RESEARCH

Comparative Study on ChatGPT and Scholars' Abstracts: Taking the Field of Information Resource Management as an Example

  • Zhang Qiang ,
  • Wang Xiaoran ,
  • Gao Ying ,
  • Wang Chang jue ,
  • Zhou Hong
Expand
  • 1 School of Humanities, Anhui Polytechnic University, Wuhu 241000;
    2 School of Information Management, Central China Normal University, Wuhan 430079;
    3 School of Computer and Information, Anhui Polytechnic University, Wuhu 241000;
    4 Department of Information Resources Management, School of Economics and Management, University of Chinese Academy of Sciences, Beijing 100191;
    5 Wuhan Library and Intelligence Center of Chinese Academy of Science, Wuhan 430071

Received date: 2023-09-05

  Revised date: 2023-12-31

  Online published: 2024-05-17

Supported by

This work is supported by Anhui Provincial teaching and Research project "Reform and Practice of Humanistic Quality Education System under the Background of New Engineering"(Grant No.2020jyxm0152) and Wuhan Literature and Information Center of Chinese Academy of Sciences Young Leading Talents Project (Grant No.E0KZ451)

Fold

Abstract

[Purpose/Significance] To explore the similarities and differences of Chinese abstracts written by ChatGPT and scholars can provide references for AI-generated academic paper detection and related research.[Method/Process] Firstly,taking the field of information resource management as an example,this paper extracted 500 highly cited papers from library science,information science,and archival science in the recent years.Based on the obtained paper titles,it used the prompt method to apply the ChatGPT tool to generate corresponding abstract texts and construct a dataset.Secondly,it adopted 9 machine learning and deep learning algorithms to classify and detect abstract texts generated by ChatGPT and written by scholars.Finally,it analyzed and revealed the similarities and differences between the two from multiple perspectives,including text features,topic models,and ROUGE evaluation.[Result/Conclusion] Mainstream machine learning and deep learning algorithms trained on datasets can effectively distinguish whether abstracts are generated by AI or written by scholars,with BERT and ERNIE performing best,while RF and Xgboost best in machine learning algorithms.The number of abstract characters and sentences generated by ChatGPT is more than that written by scholars,and the

Key words： ChatGPT; text classification; text features; paper abstract

Cite this article

Zhang Qiang , Wang Xiaoran , Gao Ying , Wang Chang jue , Zhou Hong . Comparative Study on ChatGPT and Scholars' Abstracts: Taking the Field of Information Resource Management as an Example[J]. Library and Information Service, 2024 , 68(8) : 35 -47 . DOI: 10.13266/j.issn.0252-3116.2024.08.004

References

[1] WANG F Y, LI J, QIN R, et al. ChatGPT for computational social systems:from conversational applications to human-oriented operating systems[J]. IEEE transactions on computational social systems, 2023, 10(2):414-425.
[2] MONDAL S, DAS S, VRANA V G. How to bell the cat?A theoretical teview of generative artificial intelligence towards digital disruption in all walks of life[J]. Technologies, 2023, 11(2):44.
[3] WU T, HE S, LIU J, et al. A brief overview of ChatGPT:The history, status quo and potential future development[J]. IEEE/CAA Journal of Automatica Sinica, 2023, 10(5):1122-1136.
[4] ADESHOLA I, ADEPOJU A P. The opportunities and challenges of ChatGPT in education[J/OL]. Interactive learning environments, 2023:1-14. https://doi.org/10.1080/10494820.2023.2253858.
[5] Nature. Initial submission[EB/OL].[2024-03-06]. https://www.nature.com/nature/for-authors/initial-submission.
[6] 图书情报工作. AI政策声明[EB/OL].[2024-03-06]. https://www.lis.ac.cn/CN/column/column27.shtml.(Library and information service. AI policy statement[EB/OL].[2024-03-06]. https://www.lis.ac.cn/CN/column/column27.shtml.)
[7] STOKRL-WALKER C, VAN NOORDEN R. What ChatGPT and generative AI mean for science[J]. Nature, 2023, 614(7947):214-216.
[8] BUDHWAR P, CHOWDHURY S, WOOD G, et al. Human resource management in the age of generative artificial intelligence:perspectives and research directions on ChatGPT[J]. Human resource management journal, 2023, 33(3):606-659.
[9] 戴岭,胡姣,祝智庭. ChatGPT赋能教育数字化转型的新方略[J].开放教育研究, 2023, 29(4):41-48.(DAI L, HU J, ZHU Z T. ChatGPT empowering the new strategy of digital transformation in education[J]. Open education research, 2023, 29(4):41-48.)
[10] 陆伟,刘家伟,马永强,等. ChatGPT为代表的大模型对信息资源管理的影响[J].图书情报知识, 2023, 40(2):6-9,70.(LU W, LIU J W, MA Z Q, et al. The impact of large models represented by ChatGPT on information resource management[J]. Documentation, information&knowledge, 2023, 40(2):6-9, 70.)
[11] 张智雄,于改红,刘熠,等. ChatGPT对文献情报工作的影响[J].数据分析与知识发现, 2023, 7(3):36-42.(ZHANG Z Q, YU G H, LIU Y, et al. The impact of ChatGPT on document information work[J]. Data analysis and knowledge discovery, 2023, 7(3):36-42.)
[12] LUND B D, WANG T. Chatting about ChatGPT:how may AI and GPT impact academia and libraries?[J]. Library hi tech news, 2023, 40(3):26-29.
[13] 曹树金,曹茹烨.从ChatGPT看生成式AI对情报学研究与实践的影响[J].现代情报, 2023, 43(4):3-10.(CAO S J, CAO R Y. Exploring the impact of generative AI on research and practice in information science[J]. Modern information, 2023, 43(4):3-10.)
[14] 周文欢. ChatGPT在档案领域应用和意义[J].中国档案, 2023(3):62-63.(ZHOU W H. Application and significance of ChatGPT in the field of archives[J]. China archives, 2023(3):62-63.)
[15] ZHENG H, ZHAN H. ChatGPT in scientific writing:a cautionary tale[J]. The American journal of medicine, 2023, 136(8):725-726.
[16] WALTERS W H, WILDER E I. Fabrication and errors in the bibliographic citations generated by ChatGPT[J]. Scientific Reports, 2023, 13(1):14045.
[17] 张华平,李林翰,李春锦. ChatGPT中文性能测评与风险应对[J].数据分析与知识发现, 2023, 7(3):16-25.(ZHANG H P, LI L H, LI C J. Performance evaluation and risk response of ChatGPT in Chinese[J]. Data analysis and knowledge discovery, 2023, 7(3):16-25.)
[18] 鲍彤,章成志. ChatGPT中文信息抽取能力测评——以三种典型的抽取任务为例[J].数据分析与知识发现, 2023, 7(9):1-11.(BAO T, ZHANG C Z. Evaluation of ChatGPT's Chinese information extraction ability:taking three typical extraction tasks as examples[J]. Data analysis and knowledge Discovery, 2023, 7(9):1-11.)
[19] 施亦龙,许鑫. ChatGPT机器回答与知乎人工回答的比较[J].图书馆论坛, 2024, 44(1):151-159.(SHI Y L, XU X. A comparison between ChatGPT machine answers and Zhihu artificial answers[J]. Library tribune, 2024, 44(1):151-159.)
[20] NIGH M. ChatGPT3 prompt engineering[EB/OL].[2024-03-06]. https://github.com/mattnigh/ChatGPT3-Free-Prompt-List.
[21] BLEI D M, NG A Y, JORDAN M I. Latent dirichlet allocation[J]. Journal of machine learning research, 2003, 3:993-1022.
[22] LIN C Y. ROUGE:A Package for Automatic Evaluation of Summaries[C]//Text Summarization Branches Out. Barcelona, Spain:Association for Computational Linguistics, 2004:74-81.
[23] DEVLIN J, CHANG M W, LEE K, et al. BERT:Pre-training of Deep Bidirectional Transformers for Language Understanding[C]//BURSTEIN J, DORAN C, SOLORIO T. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies, Volume 1(Long and Short Papers). Minneapolis, Minnesota:Association for Computational Linguistics, 2019:4171-4186.
[24] SUN Y, WANG S, LI Y, et al. Ernie:enhanced representation through knowledge integration[EB/OL].[2024-03-06]. https://arxiv.org/abs/1904.09223.
[25] 王红斌,金子铃,毛存礼.结合层级注意力的抽取式新闻文本自动摘要[J].计算机科学与探索, 2022, 16(4):877-887.(WANG H B, JIN Z L, MAO C L. Hierarchical attention-based extractive news text summarization[J]. Journal of frontiers of computer science and technology, 2022, 16(4):877-887.)
[26] 赵江江,王洋,许楹楹,等.基于知识蒸馏的抽取式自动摘要模型[J].计算机科学, 2023, 50(S1):214-220.(ZHAO J J, WANG Y, XU Y Y, et al. Extractive summarization model based on knowledge distillation[J]. Computer science, 2023, 50(S1):214-220.)
Options
Outlines

/

〈 〉

    Copyrights © LIS Press Co., Ltd. Reproduction is prohibited without permission.