RESEARCH PAPERS

The Performance Evaluation of Q&A Service of Library Agents: A Case Study of Tongji University “Zhi Xiaotu”

  • Liu Yueru ,
  • Chen Xin ,
  • Yao Junlan
Expand
  • Library of Tongji University, Shanghai 200092
Liu Yueru, Associate Research Librarian, Master’s Degree; Chen Xin, Library Director, Associate Research Librarian, Master’s Degree, Corresponding Author, E-mail: xchen@tongji.edu.cn; Yao Junlan, Librarian, PhD.

Received date: 2025-07-23

  Revised date: 2025-09-25

  Online published: 2026-09-24

Abstract

[Purpose/Significance] This study evaluates the performance of intelligent question-answering services provided by library agents based on different large language models (LLMs), aiming to select the most suitable model that meets the operational needs of libraries to serve as the foundation for intelligent services. [Method/Process] An evaluation system was constructed, covering four dimensions: system robustness, response timeliness, computational resource cost, and answer quality. Using the library agent “Zhi Xiaotu” of Tongji University Library as the experimental object, several mainstream domestic large language models were tested, including DeepSeek, Doubao, and Tongyi Qianwen. [Result/Conclusion] Through a multidimensional performance evaluation, this study concludes that while different LLMs exhibit different strengths in various aspects, their overall performance is comparable. To address the library’s practical requirements, a combined entropy-weight and analytic hierarchy process (AHP) approach is adopted to comprehensively score the candidate LLMs. Consequently, the most suitable model is selected and deployed to serve as the foundation for intelligent services.

Cite this article

Liu Yueru , Chen Xin , Yao Junlan . The Performance Evaluation of Q&A Service of Library Agents: A Case Study of Tongji University “Zhi Xiaotu”[J]. Library and Information Service, 2026 , 70(19) : 127 -138 . DOI: 10.13266/j.issn.0252-3116.2026.19.010

References

[1] 中国图书馆学会. 中国图书馆学会关于印发《中国图书馆学会“十四五” 发展规划纲要(2021-2025年)》的通知[EB/OL]. [2025-09-24]. https://www.lsc.org.cn/cns/html_pass/contents/1299/15358.html. (Chinese Library Association. Notice of the Chinese Library Association on issuing the “14th Five-Year” Development Plan Outline of the Chinese Library Association (2021-2025)[EB/OL]. [2025-09-24]. https://www.lsc.org.cn/cns/html_pass/contents/1299/15358.html.)
[2] 中共中央办公厅、国务院办公厅. 关于推进实施国家文化数字化战略的意见[EB/OL]. [2025-09-24]. https://www.gov.cn/zhengce/2022-05/22/content_5691759.htm. (General Office of the Communist Party of China Central Committee, General Office of the State Council. Opinions on promoting the implementation of the National Cultural Digitalization Strategy[EB/OL]. [2025-09-24]. https://www.gov.cn/zhengce/2022-05/22/content_5691759.htm.)
[3] 国家图书馆. 国家图书馆“十四五” 发展规划[EB/OL]. [2025-09-24]. https://www.nlc.cn/upload/attachments/2023-04-14/4dd2526a.pdf. (National Library of China. 14th Five-Year Development Plan of the National Library of China[EB/OL]. [2025-09-24]. https://www.nlc.cn/upload/attachments/2023-04-14/4dd2526a.pdf.)
[4] 秦小林, 古徐, 李弟诚, 等. 大语言模型综述与展望[J]. 计算机应用, 2025, 45(3): 685-696. (QIN X L, GU X, LI D C, et al. Survey and prospect of large language models[J]. Journal of computer applications, 2025, 45(3): 685-696.)
[5] SHAO M H, ABDUL B, KARRI R, et al. Survey of different large language model architectures: trends, benchmarks, and challenges [J]. IEEE transactions on access, 2024, 12: 188664-188706.
[6] 李强, 林淑凤, 于媛. 生成式人工智能赋能科技查新的实践探索与模式构建[J]. 图书馆论坛, 2025, 45(8): 107-116. (LI Q, LIN S, YU Y. Generative ai-enhanced scientific novelty search: practical exploration and framework development[J]. Library tribune, 2025, 45(8): 107-116.)
[7] 张君冬, 杨松桦, 刘江峰, 等. AIGC赋能中医古籍活化:Huang-Di大模型的构建[J]. 图书馆论坛, 2024, 44(10): 103-112. (ZHANG J, YANG S, LIU J, et al. Aaigc empowering the revitalization of ancient books on traditional Chinese medicine:building the Huang-Di large language model[J]. Library tribune, 2024, 44(10): 103-112.)
[8] 徐鸿, 宋涵, 蒋莲.基于通义千问大模型的智慧馆员问答服务系统研究[J]. 高校图书馆工作, 2024, 44(6): 58-64. (XU H, SONG H, JIANG L. Research on the intelligent librarian Q&A service system based on Tongyi Qianwen large language model[J]. Library work in colleges and universities, 2024, 44(6): 58-64.)
[9] 李丽. 大语言模型视角下DeepSeek赋能高校图书馆学科服务研究[J]. 图书馆建设, 2025(3): 2782. (LI l. DeepSeek enabling university library subject services from the perspective of large language model[J]. Library development, 2025(3): 2782.)
[10] 郭亚军, 徐苑茜, 梁艳丽, 等. 从ChatGPT到DeepSeek:生成式人工智能迭代对图书馆的影响[J]. 图书馆论坛, 2025, 45(7): 140-149. (GUO Y, XU Y, LIANG Y, et al. From ChatGPT to DeepSeek: the impact of generative artificial intelligence iterations on libraries[J]. Library tribune, 2025, 45(7): 140-149.)
[11] 罗坤明, 李子晟, 张磊, 等. 国产开源大模型赋能图书馆智慧化服务功能——以DeepSeek为例[J]. 图书馆理论与实践, 2025(5): 86-96. (LUO K M, LI Z S, ZHANG L, et al. Empowering intelligent library services with domestic open-source large models: a case study of DeepSeek [J]. Library theory and practice, 2025(5): 86-96.
[12] 杨新涯, 戴立伟, 钱国富. DeepSeek在图书馆的应用场景构架研究[J]. 图书馆论坛, 2025, 45(10): 90-97. (YANG X Y, DAI L W, QIAN G F. Research on the application scenario architecture of DeepSeek in libraries[J]. Library tribune, 2025, 45(10): 90-97)
[13] 张强, 高颖, 任豆豆, 等. 融合DeepSeek-R1和RAG技术的先秦文化元典智能问答研究[J]. 现代情报, 2026, 46(1): 173-186. (ZHANG Q, GAO Y, REN D D, et al. Research on intelligent question answering for Pre-Qin cultural classics by integrating DeepSeek-R1 and RAG technologies[J]. Journal of modern information, 2026, 46(1): 173-186.)
[14] 张强, 高颖, 辛竹琳, 等. 多模型多视角下AI生成与学者撰写文献内容的比较研究[J]. 图书馆杂志, 2026, 45(5): 37-47. (ZHANG Q, GAO Y, XIN Z, et al. Comparative study of ai-generated and scholar-written literature content under multiple models and perspectives[J]. Library journal, 2026, 45(5): 37-47.)
[15] MA H, LU Y, XIAO Z, et al. SDD-Law LLM: advancing intelligent legal systems through synthetic data-driven fine-tuning of large language models [J]. Electronics (Switzerland), 2025, 14(4): 742.
[16] TORDJMAN M, LIU Z, YUCE M, et al. Comparative benchmarking of the DeepSeek large language model on medical tasks and clinical reasoning[J]. Nature medicine, 2025, 31: 2550–2555.
[17] JI T, GRAHAM Y, JONES G J F. Contrasting human opinion of non-factoid question answering with automatic evaluation [C]//2020 Conference on Human Information Interaction and Retrieval. New York: ACM, 2020: 348-352.
[18] HIJAZI H, MOLLá D, NGUYEN V, et al. Using large language models to evaluate biomedical query-focused summarization [C]//Proceedings of the 23rd Workshop on Biomedical Language Processing, Bangkok, Thailand: Association for Computational Linguistics, 2024: 236-242.
[19] BAKIR D, AKTA? M S, YILDIZ B. A model-based evaluation metric for question answering systems [J]. International journal of software engineering and knowledge engineering, 2025, 35(2): 243-262.
[20] ZHAO W, PEYRARD M, LIU F, et al. MoverScore: text generation evaluating with contextualized embeddings and earth mover distance[C]. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Hong Kong: Association for Computational Linguistics, 2019, 563-578.
[21] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). Minneapolis: Association for Computational Linguistics, 2019: 4171-4186.
[22] ZHANG T, KISHORE V, WU F, et al. BERTSCORE: evaluating text generation with BERT [C]//Proceedings of the International Conference on Learning Representations. Addis Ababa: DBLP, 2020.
[23] PEINELT N, NGUYEN D, LIAKATA M. tBERT: topic models and BERT joining forces for semantic similarity detection [C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, 2020: 7047-7055.
[24] CHEN A, STANOVSKY G, SINGH S, et al. Evaluating question answering evaluation [C]//Proceedings of the 2nd Workshop on Machine Reading for Question Answering. Hong Kong: Association for Computational Linguistics, 2019: 119-124.
[25] 百度公司.文心ERNIE开发套件[EB/OL]. [2025-09-24]. https://ai.baidu.com/ai-doc/ERNIE-Ultimate/7l6367w4m.(Baidu Inc. ERNIE development kit[EB/OL]. [2025-09-24]. https://ai.baidu.com/ai-doc/ERNIE-Ultimate/7l6367w4m.)
[26] 火山引擎. Doubao-embedding-large— 火山方舟大模型服务平台[EB/OL]. [2025-09-24]. https://www.volcengine.com/docs/82379/1554711. (Volcengine. Doubao-embedding-large—Volcano Ark large model service platform[EB/OL]. [2025-09-24]. https://www.volcengine.com/docs/82379/1554711.)
[27] 路人与大师. BGE-M3:一个多功能、多语言、多粒度的语言向量模型[EB/OL]. [2025-09-24]. https://blog.csdn.net/weixin_41046245/article/details/142215886. (Luren Yudashi. BGE-M3: a multifunctional, multilingual, and multigranular language vector model[EB/OL]. [2025-09-24]. https://blog.csdn.net/weixin_41046245/article/details/142215886.)
[28] 北京人工智能研究院(BAAI). BGE-M3模型官方文档[EB/OL]. [2025-09-24]. https://bge-model.com/bge/bge_m3.html. (Beijing Academy of Artificial Intelligence (BAAI). Official documentation of BGE-M3 model[EB/OL]. [2025-09-24]. https://bge-model.com/bge/bge_m3.html.)
[29] 智源社区(智能研究院). 新一代通用向量模型BGE-M3:一站式支持多语言、长文本和多种检索方式[EB/OL]. [2025-09-24]. https://hub.baai.ac.cn/view/34816. (Baidu Institute of Artificial Intelligence (BAAI) Community. New-Generation general vector model BGE-M3: one-stop support for multilingual, long text and multiple retrieval methods[EB/OL]. [2025-09-24]. https://hub.baai.ac.cn/view/34816.)
[30] SATHEESH S, BECKH K, KLUG K, et al. Robustness evaluation of the German extractive question answering task[C]//Proceedings of the 31st International Conference on Computational Linguistics, COLING. Germany: Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS, 2025: 1785-1801.
[31] PENG C, GONG Z, SHEN G. Measurement and modeling of a web-based question answering system [J]. IEEE transactions on knowledge and data engineering, 2006, 18(7): 924-937.
[32] ASHRAFI T, INEZA Y, AREFIN S E, et al. Stable or shaky? The semantics of ChatGPT’s behavior under repeated queries [C]//2024 IEEE 18th International Conference on Semantic Computing (ICSC). Laguna Hills: IEEE, 2024: 110-116.
[33] 王芳, 魏中瀚, 连芷萱. 政务智能问答系统评价指标体系构建与测评问题编制[J]. 图书情报知识,2023, 40(6): 98-111. (WANG F, WEI Z H, LIAN Z X. Developing an evaluation index system and its testing questions for government intelligent Q&A systems[J]. Documentation, information & knowledge, 2023, 40(6): 98-111.)
[34] RAWAT P, MEHTA S, BAJAJ M, et al. A multi-source information retrieval system for question answering with sentiment analysis and LLM interpretation [C]//2024 International Conference on Automation and Computation (AUTOCOM). Dehradun: IEEE, 2024: 1-6.
[35] PALNISAMY A M, SANGEETHA S, NATARAJ R V. Effects of SQL query inclusive computation in application response time [J]. Journal of computational and theoretical nanoscience, 2018, 15(5): 1625-1628.
[36] HADDAD S, MOKDAD L, YOUCEF S. Response time of BPEL4WS constructors [C]//The IEEE symposium on Computers and Communications. Riccione: IEEE, 2010: 695-700.
[37] GAO M, LI M, JI T, et al. Key technologies of intelligent question-answering system for power system rules and regulations based on improved BERTserini algorithm [J]. Processes, 2024, 12(1): 58.
[38] DERIU J, RODRIGO A, OTEGI A, et al. Survey on evaluation methods for dialogue systems. Artificial intelligence review. 2021, 54(1): 755-810.
[39] SUN M T, LIU L, CHEN H. An answer quality evaluation algorithm based on semantic in community-based question answering [C]//201814th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD). Beijing: IEEE, 2018: 971-978.
[40] XIONG M X, LI A P, XIE Z S, et al. A practical approach to answer extraction for constructing QA solution [C]//2018 IEEE Third International Conference on Data Science in Cyberspace. Changsha: IEEE, 2018: 398-404.
[41] 同济大学信息办.同济大学AI应用创新平台正式上线[EB/OL]. [2025-09-24]. https://mp.weixin.qq.com/s/VkJ10cZS-j3-fVBvvoWkoQ.(Information Office of Tongji University. Tongji University AI application innovation platform officially launched[EB/OL]. [2025-09-24]. https://mp.weixin.qq.com/s/VkJ10cZS-j3-fVBvvoWkoQ.)
[42] YU M, ZHOU R, CAI Z, et al. Unravelling the relationship between response time and user experience in mobile applications[J]. Internet research, 2020, 30(5): 1353-1382.
Outlines

/

〈 〉