研究论文

知识增强大语言模型与搜索引擎用户的感知信息价值差异比较研究——基于任务复杂性与信息获取场景

  • 孙晓宁 ,
  • 张爽 ,
  • 郭文智
展开
  • 1 山西财经大学信息学院 太原 030006;
    2 山西财经大学工商管理学院 太原 030006
孙晓宁,教授,博士,博士生导师,E-mail:arborxs@163.com;张爽,硕士研究生;郭文智,博士研究生。

收稿日期: 2024-08-19

  修回日期: 2024-12-11

  网络出版日期: 2025-06-11

基金资助

本文系国家自然科学基金青年项目“人智交互情境下对话式信息搜索的用户认知加工与学习体验研究”(项目编号:72304177)研究成果之一。

A Comparative Study of Perceived Information Value Between Knowledge-Enhanced Large Language Models and Search Engine Users: Based on Task Complexity and Information Acquisition Scenarios

  • Sun Xiaoning ,
  • Zhang Shuang ,
  • Guo Wenzhi
Expand
  • 1 School of Information, Shanxi University of Finance and Economics, Taiyuan 030006;
    2 School of Business Administration, Shanxi University of Finance and Economics, Taiyuan 030006
Sun Xiaoning,professor,PhD,doctoral supervisor,E-mail:arborxs@163.com;Zhang Shuang,master candidate;Guo Wenzhi,doctoral candidate.

Received date: 2024-08-19

  Revised date: 2024-12-11

  Online published: 2025-06-11

Supported by

This work is supported by the youth project of National Natural Science Foundation of China, titled “Research on User’s Cognitive Processing and Learning Experience of Conversational Information Search in Human-AI Interaction Context” (Grant No. 72304177).

摘要

[目的/意义] 以ChatGPT、文心一言等为代表的知识增强大语言模型为用户提供一种全新的信息获取方式。与基于传统搜索引擎工具的搜索结果相比,人工智能生成内容(AIGC)的信息价值究竟表现如何,成为值得关注的一个现实问题。[方法/过程] 采用实验研究方法,设定包含简单、复杂、开放式3类复杂程度,以及旅行型、健康型、科学型3种场景的9项信息查找任务,以此探究用户分别使用文心一言、百度搜索两种工具获取信息后的感知信息价值差异。[结果/结论] 研究发现,对于两种工具而言:①用户在感知信息的“准确性”“客观性”“相关性”“有价文字”“针对性”“完整性”“参考来源或链接”上具有显著性差异,其余指标上没有显著性差异;②不同任务复杂性水平下,用户在感知信息的“语气措辞”“原创性”上没有显著性差异,在“准确性”“客观性”等11项指标上具有显著性差异;③不同信息获取场景下,用户在全部的13项感知信息价值指标上都没有显著性差异。

本文引用格式

孙晓宁 , 张爽 , 郭文智 . 知识增强大语言模型与搜索引擎用户的感知信息价值差异比较研究——基于任务复杂性与信息获取场景[J]. 图书情报工作, 2025 , 69(12) : 90 -103 . DOI: 10.13266/j.issn.0252-3116.2025.12.008

Abstract

[Purpose/Significance] Knowledge-enhanced large language models, such as ChatGPT and Wenxinyiyan, provide users with a new way to obtain information. Compared with the search results based on traditional search engine tools, how the information value of AIGC performance becomes a practical problem worthy of attention. [Method/Process] In this paper, an experimental research method was used to set up a total of nine information finding tasks containing three types of task complexity, including simple, complex and open-ended, as well as three scenarios, travelling-type, health-type, and science-type, in order to explore the difference in perceived information value after users access information using Wenxinyiyan and Baidu respectively. [Result/Conclusion] The results show that for the two tools, firstly, users have significant differences in the “accuracy”, “objectivity”, “relevance”, “valuable text”, “pertinence”, “integrity”, “reference source or link” of perceived information. No significant differences are found for the other indicators. Secondly, under different task complexity, users do not perceive significant differences in “tone wording” and “originality” of perceived information. However, significant differences are observed in the remaining 11 indicators, including “accuracy” and “objectivity.” Thirdly, under different information acquisition scenarios, there are no significant differences in all 13 perceived information value indicators.

参考文献

[1] WU T, HE S, LIU J, et al. A brief overview of ChatGPT: the history, status quo and potential future development[J]. IEEE/CAA journal of automatica sinica, 2023, 10(5): 1122-1136.
[2] 赵杨, 张雪, 范圣悦. AIGC驱动的智慧图书馆转型: 框架、路径与挑战[J]. 情报理论与实践, 2023, 46(7): 9-16. (ZHAO Y, ZHANG X, FAN S Y. AIGC-driven intelligent library transformation: framework, pathways and challenges[J]. Information studies: theory & application, 2023, 46(7): 9-16.)
[3] 郭亚军, 刘振阳, 郭一若, 等. AIGC大学生用户信息需求研究——以ChatGPT为例[J]. 情报科学, 2024, 42(8): 34-44. (GUO Y J, LIU Z Y, GUO Y R, et al. Study on the information needs of college students using AIGC: taking ChatGPT as an example[J]. Information science, 2024, 42(8): 34-44)
[4] TIBAU M, SIQUEIRA S W M, NUNES B P. ChatGPT for chatting and searching: repurposing search behavior[J]. Library & information science research, 2024, 46(4): 101331.
[5] KARUNARATNE T, ADESINA A. Is it the new Google: impact of ChatGPT on students' information search habits[C]//Proceedings of the 22nd European conference on e-learning. Pretoria: the University of South Africa, 2023: 147-155.
[6] HUANG Y, HUANG J X. Exploring ChatGPT for nextgeneration information retrieval: opportunities and challenges[J]. Web intelligence, 2024, 22(1): 31-44.
[7] 毛太田, 汤淦, 马家伟, 等. 人工智能生成内容( AIGC) 用户采纳意愿影响因素识别研究——以ChatGPT为例[J]. 情报科学, 2024, 42(7): 126-136. (MAO T T, TANG G, MA J W, et al. Factors influencing user adoption intention of artificial intelligence generated content(AIGC): a study on ChatGPT[J]. Information science, 2024, 42(7): 126-136.)
[8] MAHESHWARI G. Factors influencing students' intention to adopt and use ChatGPT in higher education: a study in the Vietnamese context[J]. Education and information technologies, 2024, 29(10): 12167-12195.
[9] MENON D, SHILPA K. “Chatting with ChatGPT”: analyzing the factors influencing users' intention to use the open AI's ChatGPT using the UTAUT model[J]. Heliyon, 2023, 9(11): e20962.
[10] CHOUDHURY A, SHAMSZARE H. Investigating the impact of user trust on the adoption and use of ChatGPT: survey analysis[J]. Journal of medical Internet research, 2023, 25: e47184.
[11] 周涛, 李松洮, 邓胜利. 用户信息搜寻转移意向研究:从搜索引擎到生成式AI[J]. 图书情报工作, 2024, 68(3): 49-58. (ZHOU T, LI S T, DENG S L. Research on users' switching intention of information seeking: from search engines to generative AI[J]. Library and information service, 2024, 68(3): 49-58.)
[12] 周涛, 吴晓颖, 邓胜利. 用户知识问答转移行为研究:从问答社区到生成式AI[J]. 情报杂志, 2024, 43(2): 200-207. (ZHOU T, WU X Y, DENG S L. Examining users' switching behavior of knowledge Q&A: from Q&A communities to generative AI[J]. Journal of intelligence, 2024, 43(2): 200-207.)
[13] 张玥, 李青宇, 刘雨琪, 等. 组态视角下AIGC应用平台用户中辍行为影响因素研究[J]. 情报理论与实践, 2024, 47(3): 130-137, 148. (ZHANG Y, LI Q Y, LIU Y Q, et al. A study on discontinuance behavior in the AIGC application platform based on the perspective of configuration[J]. Information studies: theory & application, 2024, 47(3): 130-137, 148.)
[14] 储节旺, 罗怡帆, 李佳轩. AIGC对信息生成方式及用户信息行为的影响[J]. 图书情报工作, 2023, 67(24): 13-23. (CHU J W, LUO Y F, LI J X. The influence of AIGC on information generation mode and user information behavior[J]. Library and information service, 2023, 67(24): 13-23.)
[15] 王浩伟, 汪璠, 王秉琰. 主题视角下生成式人工智能生成内容与用户生成内容的比较[J]. 情报理论与实践, 2023, 46(10): 200-207, 199. (WANG H W, WANG F, WANG B Y. A comparison of Generative AI-Generated Content and UserGenerated Content in the thematic perspective[J]. Information studies: theory & application, 2023, 46(10): 200-207, 199.)
[16] 陈隽柏, 吴国平, 张童, 等. 语言学视角下人工智能生成内容与用户生成内容的对比研究——以在线医疗服务场景为例[J]. 情报理论与实践, 2024, 47(9): 192-201. (CHEN J B, WU G P, ZHANG T, et al. Comparative study of artificial intelligence generated content and user generated content from a linguistic perspective: taking online healthcare services as an example[J]. Information studies: theory & application, 2024, 47(9): 192-201.)
[17] 施亦龙, 许鑫. ChatGPT机器回答与知乎人工回答的比较[J]. 图书 馆论 坛, 2024, 44(1): 151-159. (SHI Y L, XU X. A comparative study on the response characteristics of ChatGPT and Zhihu[J]. Library tribune, 2024, 44(1): 151-159.)
[18] AYERS J W, POLIAK A, DREDZE M, et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum[J]. JAMA internal medicine, 2023, 183(6): 589-596.
[19] 吴丹, 孙国烨. 生成式智能搜索结果可信度研究[J]. 中国图书馆学报, 2023, 49(6): 51-67. (WU D, SUN G Y. The credibility of the results of generative intelligent search[J]. Journal of library science in China, 2023, 49(6): 51-67.)
[20] 宋士杰, 赵宇翔, 朱庆华. 从ELIZA到ChatGPT:人智交互体验中的AI生成内容( AIGC)可信度评价[J]. 情报资料工作, 2023, 44(4): 35-42. (SONG S J, ZHAO Y X, ZHU Q H. From ELIZA to ChatGPT: AI-Generated Content(AIGC) credibility evaluation in human-intelligent interactive experience[J]. Information and documentation services, 2023, 44(4): 35-42.)
[21] BEDUÉ P, FRITZSCHE A. Can we trust AI? An empirical investigation of trust requirements and guide to successful AI adoption[J]. Journal of enterprise information management, 2022, 35(2): 530-549.
[22] KIM J, GIROUX M, LEE J C. When do you trust AI? The effect of number presentation detail on consumer trust and acceptance of AI recommendations[J]. Psychology & marketing, 2021, 38(7): 1140-1155.
[23] PELAU C, DABIJA D C, ENE I. What makes an AI device human-like? The role of interaction quality, empathy and perceived psychological anthropomorphic characteristics in the acceptance of artificial intelligence in the service industry[J]. Computers in human behavior, 2021, 122: 106855.
[24] 莫祖英, 盘大清, 刘欢, 等. 信息质量视角下AIGC虚假信息问题及根源分析[J]. 图书情报知识, 2023, 40(4): 32-40. (MO Z Y, PAN D Q, LIU H, et al. Analysis on AIGC false information problem and root cause from the perspective of information quality[J]. Documentation, information & knowledge, 2023, 40(4): 32-40.)
[25] BICKMORE T W, TRINH H, OLAFSSON S, et al. Patient and consumer safety risks when using conversational assistants for medical information: an observational study of Siri, Alexa, and Google assistant[J]. Journal of medical Internet research, 2018, 20(9): e11510.
[26] CHAKRABORTY C, BHATTACHARYA M, LEE S S. Need an AI-enabled, next-generation, advanced ChatGPT or large language models (LLMs) for error-free and accurate medical information[J]. Annals of biomedical engineering, 2024, 52(2): 134-135.
[27] 文心一言[EB/OL]. [2025-02-10]. http://agents.baidu.com/docs/model/ERNIE_bot_introduce/.
[28] JAEGER P T, BURNETT G. Information worlds: behavior, technology, and social context in the age of the Internet[M]. New York: Routledge, 2010.
[29] BURNETT G, LEE J, HOLLISTER J M, et al. Information value across cultures and communities[J]. Proceedings of the American Society for Information Science and Technology, 2014, 51(1): 1-4.
[30] AHITUV N, NEUMANN S. Principles of information systems for management[M]. Dubuqu: William C. Brown Publishers, 1986.
[31] WANG R Y, STRONG D M. Beyond accuracy: what data quality means to data consumers[J]. Journal of management information systems, 1996, 12(4): 5-33.
[32] LEE W Y, STRONG M D, KAHN B K, et al. AIMQ: a methodology for information quality assessment[J]. Information & management, 2002, 40(2): 133-146.
[33] MICHNIK J, LO M C. The assessment of the information quality with the aid of multiple criteria analysis[J]. European journal of operational research, 2009, 195(3): 850-856.
[34] ZHANG Y, TRACE C B. The quality of health and wellness self‐tracking data: a consumer perspective[J]. Journal of the Association for Information Science and Technology, 2022, 73(6): 879-891.
[35] HILLIGOSS B, RIEH S Y. Developing a unifying framework of credibility assessment: construct, heuristics, and interaction in context[J]. Information processing & management, 2008, 44(4): 1467-1484.
[36] OH S, WORRALL A, YI Y J. Quality evaluation of health answers in Yahoo! answers: a comparison between experts and users[J]. Proceedings of the American Society for Information Science and Technology, 2011, 48(1): 1-3.
[37] WONG W N M, CHU S K W, HUANG H, et al. Cross‐cultural quality comparison of online health information for elderly care on Yahoo! answers[J]. Proceedings of the American Society for Information Science and Technology, 2014, 51(1): 1-10.
[38] CHU S K W, HUANG H, WONG W N M, et al. Quality and clarity of health information on Q&A sites[J]. Library & information science research, 2018, 40(3/4): 237-244.
[39] ZHU L, LI H, HE W, et al. What influences online reviews’ perceived information quality? Perspectives on information richness, emotional polarity and product type[J]. The electronic library, 2020, 38(2): 273-296.
[40] SOHN D. Coping with information in social media: the effects of network structure and knowledge on perception of information value[J]. Computers in human behavior, 2014, 32: 145-151.
[41] 孙晓宁, 赵宇翔, 朱庆华. 社会化搜索平台中信息价值感知差异研究——基于用户满意度与任务复杂性视角[J]. 情报 学报, 2018, 37(1): 86-97. (SUN X N, ZHAO Y X, ZHU Q H. Perception discrepancy of information value on social search platforms: perspectives of user satisfaction and task complexity[J]. Journal of the China Society for Scientific and Technical Information, 2018, 37(1): 86-97.)
[42] RUSHO Y, RABAN D R. Hands on: information experiences as sources of value[J]. Journal of the Association for Information Science and Technology, 2020, 71(6): 671-684.
[43] PAYNE J W. Task complexity and contingent processing in decision making: an information search and protocol analysis[J]. Organizational behavior and human performance, 1976, 16(2): 366-387.
[44] GWIZDKA J, LOPATOVSKA I. The role of subjective factors in the information search process[J]. Journal of the American Society for Information Science and Technology, 2009, 60(12): 2452-2464.
[45] TOMS E G, O’BRIEN H, MACKENZIE T, et al. Task effects on interactive search: the query factor[C]//6th international workshop of the initiative for the evaluation of XML retrieval. Heidelberg: Springer, 2007: 359-372.
[46] KIM K S, ALLEN B. Cognitive and task influences on web searching behavior[J]. Journal of the American Society for Information Science and Technology, 2002, 53(2): 109-119.
[47] QIU L. Analytical searching vs. browsing in hypertext information retrieval systems[J]. Canadian journal of information and library science, 1993, 18(4): 1-13.
[48] CHIN J, FU W T, KANNAMPALLIL T. Adaptive information search: age-dependent interactions between cognitive profiles and strategies[C]//Proceedings of the SIGCHI conference on human factors in computing systems. Boston: ACM, 2009: 1683-1692.
[49] LIU J, KIM C S, CREEL C. Exploring search task difficulty reasons in different task types and user knowledge groups[J]. Information processing & management, 2015, 51(3): 273-285.
[50] BYSTRÖM K, JÄRVELIN K. Task complexity affects information seeking and use[J]. Information processing & management, 1995, 31(2): 191-213.
[51] BELL D J, RUTHVEN I. Searcher’s assessments of task complexity for web searching[C]//26th European conference on IR research. Heidelberg: Springer, 2004: 57-71.
[52] BARSKY E, BAR-ILAN J. The impact of task phrasing on the choice of search keywords and on the search process and success[J]. Journal of the American Society for Information Science and Technology, 2012, 63(10): 1987-2005.
[53] 廉同辉, 余菜花. 基于任务复杂度、时间限制和搜寻能力的游客社会化搜寻效果影响机制研究[J]. 情报杂志, 2020, 39(12): 151-156. (LIAN T H, YU C H. The influence mechanism of tourists' socialized search results based on task complexity, time limit and search ability[J]. Journal of intelligence, 2020, 39(12): 151-156.)
[54] WALHOUT J, OOMEN P, JARODZKA H, et al. Effects of task complexity on online search behavior of adolescents[J]. Journal of the Association for Information Science and Technology, 2017, 68(6): 1449-1461.
[55] 姜婷婷, 贺虹虹, 张正楠. 搜索任务复杂度对用户情感的影响研究[J]. 图书情报知识, 2016(4): 74-82. (JIANG T T, HE H H, ZHANG Z N. Impacts of search task complexity on users' emotions[J]. Documentation, information & knowledge, 2016(4): 74-82.)
[56] BEHZADI H, SANATJOO A, FATTAHI R, et al. Survey the role of emotions in information retrieval[J]. Iranian journal of information processing and management, 2016, 31(2): 531-553.
[57] ARAPAKIS I, JOSE J M, GRAY P D. Affective feedback: an investigation into the role of emotions in the information seeking process[C]//Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval. New York: ACM, 2008: 395-402.
[58] SAASTAMOINEN M, JÄRVELIN K. Search task features in work tasks of varying types and complexity[J]. Journal of the Association for Information Science and Technology, 2017, 68(5): 1111-1123.
[59] TOMS E G, FREUND L, KOPAK R, et al. The effect of task domain on search[C]//Proceedings of the 2003 conference of the centre for advanced studies on collaborative research. Toronto: IBM, 2003.
[60] TSAI M J, LIANG J C, HOU H T, et al. University students' online information searching strategies in different search contexts[J]. Australasian journal of educational technology, 2012, 28(5): 881-895.
[61] RIEH S Y. Credibility assessment of online information in context[J]. Journal of information science theory and practice, 2014, 2(3): 6-17.
[62] 陈璟浩, 罗友渔, 杨超然, 等. 突发公共卫生事件情境下视障人士的信息搜寻行为研究——基于情境任务型实验方法[J]. 情报科学, 2024, 42(1): 122-131, 152. (CHEN J H, LUO Y Y, YANG C R, et al. The information-seeking behavior of visually impaired people in the context of public health emergencies: a method of contextual task-based experimentation[J]. Information science, 2024, 42(1): 122-131, 152.)
[63] SURI S, COUNTS, S, WANG L, et al. The use of generative search engines for knowledge work and complex tasks[EB/OL]. [2025-02-12]. https://arxiv.org/abs/2404.04268.
[64] XU R, FENG Y, CHEN H. ChatGPT vs. Google: a comparative study of search performance and user experience[EB/OL]. [2025-02-12]. https://arxiv.org/abs/2307.01135.
[65] SONG X, LIU C, ZHANG Y. Chinese college students’ source selection and use in searching for health-related information online[J]. Information processing & management, 2021, 58(3): 102489.
[66] NAZI Z A, PENG W. Large language models in healthcare and medical domain: a review[J]. Informatics, 2024, 11(3): 57.
[67] GUO B, ZHANG X, WANG Z, et al. How close is ChatGPT to human experts? Comparison corpus, evaluation, and detection[EB/OL]. [2025-02-12]. https://arxiv.org/abs/2301.07597.
[68] WU J, GAN W, CHEN Z, et al. AI-Generated Content (AIGC): a survey[EB/OL]. [2025-02-12]. https://arxiv.org/abs/2304.06632.
[69] 孙晓宁, 赵宇翔, 朱庆华. 基于SQA系统的社会化搜索答案质量评价指标构建[J]. 中国图书馆学报, 2015, 41(4): 65-82. (SUN X N, ZHAO Y X, ZHU Q H. The construction of answer quality evaluation index for social search: an empirical study on social Q&A system[J]. Journal of library science in China, 2015, 41(4): 65-82.)
[70] SULLIVAN G M, ARTINO A R. Analyzing and interpreting data from Likert-type scales[J]. Journal of graduate medical education, 2013, 5(4): 541-542.
[71] NORMAN G. Likert scales, levels of measurement and the “laws” of statistics[J]. Advances in health sciences education: theory and practice, 2010, 15(5): 625-632.
[72] 梁怀新, 宋诚.AIGC时代的网络信息内容生态安全风险及其治理——兼以ChatGPT为对象的实验访谈案例分析[J]. 图书情报工作, 2023, 67(20): 58-69. (LIANG H X, SONG C. Ecological security risk and governance of network information content in AIGC era: take ChatGPT as an example[J]. Library and information service, 2023, 67(20): 58-69.)
文章导航

/

〈 〉