Library and Information Service

Library and Information Service >

2026 , Vol. 70 >Issue 4: 116 - 127

DOI: https://doi.org/10.13266/j.issn.0252-3116.2026.04.009

RESEARCH PAPERS

Analysis of the Feature Differences between AI Texts and Scholar-written Texts: A Case Study of Highly Cited Papers in the Field of Higher Education Studies

  • Wang Weizheng ,
  • Qiao Hong ,
  • Ding Xiaoyan
Expand
  • 1 Shandong Normal University Library, Jinan 250358;
    2 Shandong Normal University Business School, Jinan 250358

Received date: 2024-12-27

  Revised date: 2025-03-11

  Online published: 2026-02-12

Supported by

This work is supported by the Social Science Planning Research Project of Shandong Province titled “Research on Information Sharing and User Engagement from the Perspective of Smart Libraries” (Grant No. 23DTQJ04).

Fold

Abstract

[Purpose/Significance] This study explores the characteristic distinctions between AI-generated and scholar-written texts and develops an AI text detector to promote a sustainable balance between technological advancement and human intellectual contribution in the educational sector. [Method/Process] Utilizing a sample of 98 highly cited papers in the field of higher education studies, abstracts were generated by GPT-4 based on paper titles, forming a detection dataset. Machine learning and deep learning algorithms were employed to differentiate AI-generated texts from human-authored ones, while a multidimensional analysis was conducted on text characteristics and generative quality assessments. [Result/Conclusion] The detection models based on machine learning and deep learning demonstrates a high efficacy in identifying AI-generated abstracts. AI-generated texts tend to exhibit a positive deviation in length, with more generalized and macro-level vocabulary choices. Furthermore, they display certain cultural and linguistic inconsistencies and exhibit more uniform sentence structures. In contrast, scholar-written abstracts are characterized by greater conciseness, flexibility, and depth of thought, showing a closer alignment with China’s specific policy contexts and educational realities.

Key words： artificial intelligence generated content; AI text detection; text classification; abstract detection

Cite this article

Wang Weizheng , Qiao Hong , Ding Xiaoyan . Analysis of the Feature Differences between AI Texts and Scholar-written Texts: A Case Study of Highly Cited Papers in the Field of Higher Education Studies[J]. Library and Information Service, 2026 , 70(4) : 116 -127 . DOI: 10.13266/j.issn.0252-3116.2026.04.009

References

[1] HU K. ChatGPT sets record for fastest-growing user base - analyst note[EB/OL]. [2025-02-15]. https://www.nasdaq.com/articles/chatgpt-sets-record-for-fastest-growing-userbase-analystnote.
[2] DERGAA I, CHAMARI K, ZMIJEWSKI P, et al. From human writing to artificial intelligence-generated text: examining the prospects and potential threats of ChatGPT in academic writing[J]. Biology of sport, 2023, 40(2): 615-622.
[3] RASUL T, NAIR S, KALENDRA D, et al. The role of ChatGPT in higher education: benefits, challenges, and future research directions[J]. Journal of applied learning and teaching, 2023, 6(1): 41-56.
[4] WANG F Y, LI J, QIN R, et al. ChatGPT for computational social systems: from conversational applications to human-oriented operating systems[J]. IEEE transactions on computational social systems, 2023, 10(2): 414-425.
[5] WU T, HE S, LIU J, et al. A brief overview of ChatGPT: the history, status quo, and potential future development[J]. IEEE/CAA journal of automatica Sinica, 2023, 10(5): 1122-1136.
[6] TANGERMANN V. 89 Percent of college students admit to using ChatGPTfor homework, study claims wait, what!?[EB/OL]. [2025-02-15]. https://futurism.com/the-byte/students-admitchatgpt-homework.
[7] 龚芙蓉. ChatGPT类生成式AI对高校图书馆数字素养教育的影响探析[J]. 图书情报知识, 2023, 40(5): 97-106, 156. (GONG F R. The impact of generative AI like ChatGPT on digital literacy education in university libraries[J]. Documentation, information & knowledge, 2023, 40(5): 97-106, 156.)
[8] 尚智丛, 闫禹宏. ChatGPT教育应用及其带来的变革与伦理挑战[J]. 东北师大学报( 哲学社会科学版), 2023(5): 44-54. (SHANG Z C, YAN Y H. ChatGPT education application and its changes and ethical challenges[J]. Journal of Northeast Normal University (philosophy and social sciences edition), 2023(5): 44-54.)
[9] 钟秉林, 尚俊杰, 王建华, 等. ChatGPT对教育的挑战( 笔谈) [J]. 重庆高教研究, 2023, 11(3): 3-25. (ZHONG B L, SHANG J J, WANG J H, et al. ChatGPT 's challenges to education[J]. Chongqing higher education research, 2023, 11(3): 3-25.)
[10] NATURE. Initial submission[EB/OL]. [2025-05-05]. https://www.nature.com/nature/for-authors/initial-submission.
[11] 图书情报工作. 《图书情报工作》杂志社关于AI生成论文内容的政策声明[EB/OL]. [2025-05-05]. https://www.lis.ac.cn/CN/news/news251.shtml. (Library and information service.AI policy statement[EB/OL]. [2025-05-05]. https://www.lis.ac.cn/CN/news/news251.shtml.)
[12] 储节旺，杜秀秀. 生成式人工智能赋能科研知识生产研究述评[J]. 大学图书馆学报, 2024, 42(3): 108-117. (CHU J W, DU X X. Review of research on generative artificial intelligence empowering scientific knowledge production[J]. Journal of university library, 2024(3): 108-117.)
[13] RUDOLPH J, TAN S, TAN S. ChatGPT: bullshit spewer or the end of traditional assessments in higher education[J]. Journal of applied learning and teaching, 2023, 6(1): 342-363.
[14] 叶鹰, 朱秀珠, 魏雪迎, 等. 从ChatGPT爆发到GPT技术革命的启示[J]. 情报理论与实践, 2023, 46(6): 33-37. (YE Y, ZHU X Z, WEI X Y, et al. Enlightenment from the explosion of ChatGPT to the technological revolution of GPT[J]. Information studies: theory & application, 2023, 46(6): 33-37.)
[15] 杜春娟. ChatGPT语言模型对高校师生关系的挑战与应对 —— 基于角色理论的分析[J]. 教育理论与实践, 2024, 44(21): 3-8. (DU C J. ChatGPT language model challenges and responds to the relationship between teachers and students in colleges and universities: an analysis based on role theory[J]. Educational theory and practice, 2024, 44(21): 3-8.)
[16] PERKINS M. Academic integrity considerations of AI large language models in the post-pandemic era: ChatGPT and beyond[J]. Journal of university teaching and learning practice, 2023, 20(2): 3-6.
[17] HUGLE T. The wide range of opportunities for large language models such as ChatGPT in rheumatology[J]. RMD open, 2023, 9(2): 2-4.
[18] DAY T. A preliminary investigation of fake peer-reviewed citations and references generated by ChatGPT[J]. The professional geographer, 2023, 75(6): 1024-1027.
[19] CHEN Z, CHEN C, YANG G, et al. Research integrity in the era of artificial intelligence: challenges and responses[J]. Medicine, 2024, 103(27): e38811.
[20] VAN E A M, BOLLEN J, ZUIDEMA W, et al. ChatGPT: five priorities for research[J]. Nature, 2023, 614(7947): 224-226.
[21] BAI L, LIU X, SU J. ChatGPT: the cognitive effects on learning and memory[J]. Brain‐x, 2023, 1(3): e30.
[22] 王树义, 张庆薇. ChatGPT给科研工作者带来的机遇与挑战[J]. 图书馆论坛, 2023, 43(3): 109-118. (WANG S Y, ZHANG Q W. ChatGPT’s opportunities and challenges for researchers[J]. Library tribune, 2023, 43(3): 109-118.)
[23] 赵杨, 张雪, 范圣悦. AIGC驱动的智慧图书馆转型：框架、路径与挑战[J]. 情报理论与实践, 2023, 46(7): 9-16. (ZHAO Y, ZHANG X, FAN S Y. AIGC-driven intelligent library transformation: framework, pathways and challenges[J]. Information studies: theory & application, 2023, 46(7): 9-16)
[24] FIRAT M. What ChatGPT means for universities: perceptions of scholars and students[J]. Journal of applied learning and teaching, 2023, 6(1): 1-3.
[25] 董储超, 刘旭东. 人工智能参与学位论文创作的法律规制[J]. 湖南师范大学教育科学学报, 2024, 23(6): 83-90. (DDONG C C, LIU X D. Legal regulation of artificial intelligence participation in dissertation creation[J]. Journal of education science of Hunan Normal University, 2024, 23(6): 83-90.)
[26] GIMPEL H, HALL K, DECKER S, et al. Unlocking the power of generative AI models and systems such as GPT-4 and ChatGPT for higher education: a guide for students and lecturers[R]. Stuttgart: University of Hohenheim, 2023.
[27] 张强, 王潇冉, 高颖, 等. ChatGPT生成与学者撰写文献摘要的对比研究—— 以信息资源管理领域为例[J]. 图书情报工作, 2024, 68(8): 35-47. (ZHANG Q, WANG X R, GAO Y, et al. Comparative study on ChatGPT and scholars' abstracts: taking the field of information resource management as an example[J]. Library and information service, 2024, 68(8): 35-47.)
[28] 郭鑫, 王一博, 王继民. ChatGPT生成中文学术内容分析—— 以情报学领域为例[J]. 图书馆论坛, 2024, 44(3): 134-143. (GUO X, WANG Y B, WANG J M. Feature analysis of Chinese academic content generated by ChatGPT: taking the field of intelligence as an example[J]. Library tribune, 2024, 44(3): 134- 143.)
[29] 陈隽柏, 吴国平, 张童, 等. 语言学视角下人工智能生成内容与用户生成内容的对比研究—— 以在线医疗服务场景为例[J]. 情报理论与实践, 2024, 47(9): 192-201. (CHEN J B, WU G P, ZHANG T, et al. Comparative study of artificial intelligence generated content and user generated content from a linguistic perspective： taking online healthcare services as an example[J]. Information studies: theory & application, 2024, 47(9): 192-201.)
[30] CHAKA C. Detecting AI content in responses generated by ChatGPT, YouChat, and Chatsonic: the case of five AI content detection tools[J]. Journal of applied learning and teaching, 2023, 6(2): 2-4.
[31] DIKILITAS K, KLIPPEN M I F, KELES S. A systematic rapid review of empirical research on students’ use of ChatGPT in higher education[J]. Nordic journal of systematic reviews in education, 2024, 2(1): 103-125.
[32] DEVLIN J. Bert: pre-training of deep bidirectional transformers for language understanding[EB/OL]. [2025-02-15]. https://doi. org/10.48550/arXiv.1810.04805.
[33] SUN Y, WANG S, LI Y, et al. Ernie: enhanced representation through knowledge integration[EB/OL]. [2025-02-15]. https://doi.org/10.48550/arXiv.1904.09223.
[34] HEARST M A, DUMAIS S T, OSUNA E, et al. Support vector machines[J]. IEEE intelligent systems and their applications, 1998, 13(4): 18-28.
[35] BREIMAN L. Random forests[J]. Machine learning, 2001, 45: 5-32.
[36] MARON M E, KUHNS J L. On relevance, probabilistic indexing and information retrieval[J]. Journal of the ACM, 1960, 7(3): 216-244.
[37] BISHOP C M, NASRABADI N M. Pattern recognition and machine learning[M]. New York: Springer, 2006.
[38] KE G, MENG Q, FINLEY T, et al. Lightgbm: a highly efficient gradient boosting decision tree[J]. Advances in neural information processing systems, 2017, 30(9): 3149-3157.
[39] ACHIAM J, ADLER S, AGARWAL S, et al. Gpt-4 technical report[EB/OL]. [2025-02-15]. https://doi.org/10.48550/arXiv.2303.08774.
[40] WANG M, WANG M, XU X, et al. Unleashing ChatGPT's power: a case study on optimizing information retrieval in flipped classrooms via prompt engineering[J]. IEEE transactions on learning technologies, 2023, 17(10): 629-641.
[41] SUHUTZE H, MANING C D, RAGHAVAN P. Introduction to information retrieval[M]. Cambridge: Cambridge University Press, 2008.
[42] JACCARD P. The distribution of the flora in the alpine zone[J]. New phytologist, 1912, 11(2): 37-50.
[43] 钟祥铭, 方兴东, 顾烨烨. ChatGPT的治理挑战与对策研究—— 智能传播的“ 科林格里奇困境” 与突破路径[J]. 传媒观察, 2023(3): 25-35. (ZHONG X M, FANG X D, GU Y Y. Research on governance challenges and countermeasures of ChatGPT: 'Collingridge dilemma' and breakthrough path of intelligent communication[J]. Media observation, 2023(3): 25-35.)
Options
Outlines

/

〈 〉

    Copyrights © LIS Press Co., Ltd. Reproduction is prohibited without permission.