[Purpose/Significance] Large language models, represented by GPT, excel in context understanding and reasoning. They can analyze text to discern opinions, emotions, and themes, and possess strong capabilities in content comprehension and generation. Current evaluation of academic papers heavily relies on subjective assessments, such as peer review and editorial commentary, leading to a rigid evaluation model. Therefore, exploring the potential of large language models for objective paper review is worth considering. This could help to more accurately reflect the quality of the papers and enrich the evaluation mechanisms. [Method/Process] First, this study analyzed the historical evolution and core tasks of academic paper evaluation. Building on it, by dissecting the core technology and natural language processing capabilities of large language models, it identified four potential benefits for academic paper evaluation, and conducted evaluation tests using GPT-4. Finally, it proposed an academic paper evaluation framework that integrates large language models, and analyzed the associated challenges and risks. [Results/Conclusion] Large language models have the potential to transform and develop the mechanism for evaluating academic papers. However, continuous technological upgrades and model improvements are necessary to address the challenges and risks associated with their application.
[1] 科技评价体系建设如何破局? [N]. 人民政协报, 2022-12-1(7). (How to break the situation in the construction of science and technology evaluation system? [N]. CPPCC DAILY, 2022-12-1(7).)
[2] GOERTZEL B. Artificial general intelligence: concept, state of the art, and future prospects[J]. Journal of artificial general intelligence, 2014, 5(1): 1-48.
[3] WEI J, TAY Y, BOMMASANI R, et al. Emergent abilities of large language models[J/OL]. [2024-02-09]. https://api.semanticscholar.org/CorpusID:5959482.
[4] 陆伟, 刘家伟, 马永强, 等. ChatGPT为代表的大模型对信息资源管理的影响[J]. 图书情报知识, 2023, 40(2): 6-9. (LU W, LIU J W, MA Y Q, et al. The influence of large language models represented by ChatGPT on information resources management [J]. Documentation, information & knowledge, 2023, 40(2): 6-9.)
[5] 张耀铭. 欧美同行评议的历史演进与发展趋势[J]. 昆明学院学报, 2023, 45(1): 23-30. (ZHANG Y M. The historical evolution and trend of peer review among European and American peers[J]. Journal of Kunming University, 2023, 45(1): 23-30.)
[6] GARGIELD E. Citation Indexes for Science: a new dimension in documentation through association of ideas[J]. Science, 1955, 122(3159): 108-111.
[7] Science Citation Index Expanded. Web of Science Group[EB/OL]. [2024-03-26]. https://clarivate.com/webofsciencegroup/.
[8] MA N, GUAN J, ZHAO Y. Bringing PageRank to the citation analysis[J]. Information processing & management, 2008, 44(2): 800-810.
[9] KLEINBERG J M. Authoritative sources in a hyperlinked environment[J]. Journal of the ACM, 1999, 46(5): 604-632.
[10] ADIE E. The rise of altmetrics[J]. Altmetrics: a practical guide for librarians, researchers and academics, 2016, 35(4): 67-82.
[11] LINDSAY, J MICHAEL. PlumX from plum analytics: not just altmetrics[J]. Journal of electronic resources in medical libraries, 2016, 13(1): 8-17.
[12] 默顿. 科学社会学[M]. 鲁旭东, 林聚任, 译. 北京: 商务印书馆, 2003: 610-611. (MERTON R K. The sociology of science[M]. LU X D, LIN J R, trans. Beijing: Commercial Press, 2003: 610-611.)
[13] MIKOLOV T, CHEN K, CORRADO G, et al. Efficient estimation of word representations in vector space[J/OL]. [2024-04-09]. http://arxiv.org/pdf/1301.3781.pdf.
[14] PENNINGTON J, SOCHER R, MANNING C D. Glove: global vectors for word representation[C]//Proceedings of the 2014 conference on empirical methods in natural language processing. Qatar: Association for Computational Linguistics, 2014: 1532-1543.
[15] VENKATESAN R, LI B. Convolutional neural networks in visual computing: a concise guide [M]. Boca Raton: CRC Press, 2017.
[16] VASWANI A, SHAZEER N, PARMARN, et al. Attention is all you need[J/OL]. [2024-03-29]. https://doi.org/10.48550/arXiv.1706.03762.
[17] RADFORD A, Wu J, AMODEI D, et al. Better language models and their implications[EB/OL]. [2024-02-17]. https://medium.com/@eraiitk/better-language-models-and-their-implications-d4d1cab162e6.
[18] DEVLIN J, CHANG M W, LEE K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[C]//BURSTEIN J. Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics: human language technologies. Minneapolis: Association for Computational Linguistics, 2018: 4171-4186.
[19] RADFORD A, NARASIMHAN K, SALIMANS T, et al. Improving language understanding by generative pre-training[EB/OL]. [2024-03-11]. https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf
[20] STIENNON N, OUYANG L, Wu J, et al. Learning to summarize with human feedback[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. New York: Curran Associates Inc, 2020: 3008-3021.
[21] HECHT-NIELSEN R. Theory of the backpropagation neural network[M]. Neural networks for perception. New York: Academic Press, 1992: 65-93.
[22] OPEN AI. GPT-4 Technical Repor[EB/OL]. [2023-10-23]. https://cdn.openai.com/papers/gpt-4.pdf.
[23] KUSNER M, SUN Y, KOLKINN, et al. From word embeddings to document distances[C]//Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37. International conference on machine learning. Lille: JMLR.org, 2015: 957-966.
[24] ZHANG H, GOODFELLOW I, METAXAS D, et al. Self-attention generative adversarial networks[C]//Chaudhuri.Proceedings of the 36th International Conference on Machine Learning. New York: PMLR, 2019: 7354-7363.
[25] LIU J, SHEN D, ZHANG Y, et al. What Makes Good In-Context Examples for GPT-3[C]// Proceedings of deep learning inside out. Dublin: ACI, 2022: 100-114.
[26] 杜向民, 樊建强. 人文社科类学术成果同行评议的指标体系及权重分配[J]. 中国高教研究, 2015(1): 70-74, 101. (DU X M, FAN J Q. The index system and weight distribution of peer review in academic achievements of humanities and social sciences[J]. China higher education research, 2015(1): 70-74, 101.)
[27] 邱均平. 人文社会科学评价理论与实践(上、下册)[M]. 武汉: 武汉大学出版社, 2012. (QIU J P. Theory and practice of evaluation in humanities and social sciences (volume I & II) [M]. Wuhan: Wuhan University Press, 2012.)
[28] 杜杏叶. 学术论文关键指标智能化评价研究[D]. 吉林: 吉林大学, 2019. (DU X Y. Research on intelligent evaluation of academic papers’ key factors[D]. Jilin: Jilin University, 2019.)
[29] HEINZE T, SHAPIRA P, ROGERS J, et al. Creativity capabilities and the conduct of highly innovative research in Europe and the United States[EB/OL]. [2024-03-30]. Creativity capabilities and the conduct of highly innovative research in Europe and the United States|CREA|Project|Fact sheet|FP6|CORDIS|European Commission (europa.eu).
[30] 李存娜. 人文社会科学评价的问题与反思[J]. 学术界, 2004(3): 154-167. (LI C N. Issues and reflections on evaluation of humanities and social sciences[J]. Academics in China, 2004(3): 154-167.)
[31] 何川. 科技期刊论文质量的评价指标与方法[C]//四川省科技期刊编辑学会.科技编辑出版研究文集(第八集). 四川:《电子对抗技术》编辑部, 2005: 86-89. (HE C. Evaluation indicators and methods for the quality of scientific and technological journal articles [C]//Sichuan science and technology journal editors society. Research collection of science and technology editors and publishing (Volume 8). Sichuan: “Electronic countermeasures technology” editorial department, 2005: 86-89.)
[32] 赵均, 李波. 学术期刊审稿视角下论文定性评价指标分析[J]. 中国出版, 2022(16): 19-23. (ZHAO J, LI B. Analysis of qualitative evaluation indicators of papers from the perspective of academic journal reviewers [J]. China academic journal electronic publishing house, 2022(16): 19-23.)
[33] 王辉, 黄晓林, 蒋欣宏. 科技报告文献质量评价体系构建及实证研究[J]. 湘潭大学学报(哲学社会科学版), 2021, 45(5): 188-193. (WANG H, HUANG X L, JIANG X H. Construction of and empirical study on the quality evaluation system of scientific and technical report literature[J]. Journal of Xiangtan University (philosophy and social sciences), 2021, 45(5): 188-193.)
[34] STOKEL-WALKER C. AI Chatbots are coming to search engines-can you trust the results?[EB/OL]. [2024-01-25]. https://www.nature.com/articles/d41586-023-00423-4.
[35] 彭泽, 叶光辉, 毕崇武. 学术文献中知识单元的关系计量研究[J]. 现代情报, 2023, 43(6): 56-66. (PENG Z, YE G H, BI C W. Relationship measurement of knowledge elements in academic literature[J]. Journal of modern information, 2023, 43(6): 56-66.)
[36] 李家深. 面向科技评估的专家评审系统研究与实践[J]. 情报工程, 2022, 8(2): 119-125. (LI J S. Research and practice of expert evaluation system for science and technology evaluation [J]. Technology intelligence engineering, 2022, 8(2): 119-125.)
