RESEARCH PAPERS

Research on the Experience and Performance Differences of College Students’ Informal Learning Behavior in Multimodal Human-Computer Interaction Scenarios

  • Su Wencheng ,
  • Ju Zixuan ,
  • Pan Ying ,
  • Amalia Isabel Rivero Chávez ,
  • Sun Yinglin
Expand
  • 1 Institute of Science and Technology Information, Jiangsu University, Zhenjiang 212013;
    2 Jiangsu University Library, Zhenjiang 212013;
    3 Jiangsu University, Management School, Zhenjiang 212013;
    4 Jiangsu University, School of Intellectual Property, Zhenjiang 212013
Su Wencheng, Associate Research Librarian, PhD, E-mail: bao5ye@ujs.edu.cn; Ju Zixuan, Master’s Candidate; Pan Ying, Associate Research Librarian, Master; Amalia Isabel Rivero Chávez, Doctoral Candidate; Sun Yinglin, Lecturer, PhD.

Received date: 2025-01-03

  Revised date: 2025-03-11

  Online published: 2026-03-04

Supported by

This work is supported by the Youth Project of National Social Science Fund of China, titled “Research on the Mechanism and Path of Library Space Reshaping from the Perspective of Embodied Cognition”(Grant No.22CTQ015); the Jiangsu Provincial Graduate Research and Practice Innovation Program titled “Research on the Mechanism of User Experience Influence in Multimodal Human-AI Interaction” (Grant No.SJCX24_2384); and the key project of Jiangsu Provincial Academic Library Committee’s Education Reform Research Program, titled “Research on the Principles and Strategies for Constructing Future Learning Centers in the Context of the ‘Artificial Intelligence+’ Higher Education Ecosystem” (Grant No.2024JTZD04).

Abstract

[Purpose/Significance] This study aims to explore the differences in informal learning behaviors among university students in a multimodal human-computer interaction context, focusing on enhancing user experience and performance, through a between-group comparative experiment. [Method/Process] Focusing on informal learning, this study designed evaluation metrics for text, voice, and virtual characters of artificial intelligence, from a unimodal text interface to a multimodal virtual character with voice interaction and Q&A. Using experimental methods such as eye-tracking and skin electrophysiological signals, it used statistical methods such as one-way ANOVA and non-parametric tests to study the informal learning behavior of college students. [Result/Conclusion] The study finds significant differences in visual attention and emotional responses among university students in the text-only, audio-only, and virtual character groups with higher user experience and lower learning performance in the multimodal virtual character group. This study provides an in-depth analysis of university students’ informal learning behaviors in multimodal human-computer interaction contexts and proposes strategies to enhance user experience and learning performance. Additionally, the study offers practical recommendations for the application of multimodal interactions in educational service domains such as libraries.

Cite this article

Su Wencheng , Ju Zixuan , Pan Ying , Amalia Isabel Rivero Chávez , Sun Yinglin . Research on the Experience and Performance Differences of College Students’ Informal Learning Behavior in Multimodal Human-Computer Interaction Scenarios[J]. Library and Information Service, 2026 , 70(5) : 104 -115 . DOI: 10.13266/j.issn.0252-3116.2026.05.009

References

[1] 胡国良, 黄美初. “5G+AI”视域下智慧学习空间的构建研究——基于开放大学的实践探索[J]. 远程教育杂志, 2020, 38(3): 95-104. (HU G L, HUANG M C. Research on the construction of smart learning space in the perspective of “5G+AI”: based on the practical exploration of open university[J]. Journal of distance education, 2020, 38(3): 95-104.)
[2] 武法提, 杨重阳, 李坦.智慧学习环境中的人机协同设计[J]. 电化教育研究, 2024, 45(2): 84-90. (WY F T, YANG C Y, LI T. Design of human-machine collaboration in smart learning environment[J]. E-education research, 2024, 45(2): 84-90.)
[3] 蔡昆.终身学习视域下智慧学习空间建设思路——以开放大学办学体系实践为例[J]. 贵州开放大学学报, 2023, 31(4): 1-6. (CAI K. Smart learning space construction ideas in the perspective of lifelong learning: taking the practice of open university system as an example[J]. Journal of Guizhou Open University, 2023, 31(4): 1-6.)
[4] 余胜泉, 毛芳.非正式学习——e-Learning研究与实践的新领域[J]. 电化教育研究, 2005(10): 19-24. (YU S Q, MAO F. Informal learning: a new field in e-learning research and practice[J]. E-education research, 2005(10): 19-24.)
[5] 张伟平, 马培峰.非正式学习中个人隐性知识的构建[J]. 湘潭师范学院学报(社会科学版), 2007(6): 175-176. (ZHANG W P, MA P F. The construction of personal tacit knowledge in informal learning[J]. Journal of Xiangtan Normal University(social science edition), 2007(6): 175-176.)
[6] 王芮.研究生的非正式学习行为调查研究[D]. 上海:上海师范大学, 2011. (WANG R. The investigation into the informal learning behavior of graduate student[D]. Shanghai: Shanghai Normal University, 2011.)
[7] 臧峰, 何京蕴.连续的学习区——激发非正式学习行为的空间设计策略[J]. 新建筑, 2023(5): 11-16. (ZANG F, HE J Y. Continuous learning zones: catalyzing informal learning through spatial design strategies[J]. New architecture, 2023(5): 11-16.)
[8] DOWNES S. New technology supporting informal learning[J]. Journal of emerging technologies in Web intelligence, 2010, 2(1): 27-33.
[9] HAGER P. Recognition of informal learning: challenges and issues[J]. Journal of vocational education and training, 1998, 50(4): 521-535.
[10] 蔡晴婷, 袁小娟, 徐佳宁, 等. 智慧校园下非正式学习空间建设现状及对策探析[J]. 湖北经济学院学报(人文社会科学版), 2023, 20(7): 62-66. (CAI Q T, YUAN X J, XU J N, et al. Analysis of the current situation and countermeasures for constructing informal learning spaces in smart campuses[J]. Journal of Hubei University of Economics (humanities and social sciences), 2023, 20(7): 62-66.)
[11] 徐沐阳.教育4.0背景下的未来校园非正式学习空间探析[J]. 建筑与文化, 2024(3): 36-38. (XU M Y. A exploration of informal learning spaces in future campus under the background of education 4.0[J]. Architecture and culture, 2024(3): 36-38.)
[12] DUGDALE S, 李苏萍.非正式学习图景的规划策略[J]. 住区, 2015(2): 8-27. (DUGDALE S, LI S P. Planning strategies for the informal learning landscape[J]. Design community, 2015(2): 8-27.)
[13] 季娇, 伍新春, 青紫馨.非正式学习:学习科学研究的生长点[J]. 北京师范大学学报(社会科学版), 2017(1): 74-82. (JI J, WU X C, QING Z X. Informal learning: a growing research field of learning sciences[J]. Journal of Beijing Normal University (social sciences), 2017(1): 74-82.)
[14] 吴颖.国内非正式学习研究的回顾与展望[J]. 河北大学成人教育学院学报, 2020, 22(1): 48-56. (WU Y. Review and prospect of informal learning research in China[J]. Journal of Adult Education College of Hebei University, 2020, 22(1): 48-56.)
[15] 许浩, 程卿玄, 董晶, 等. 人智交互中的AI世代:缘起、特征与未来展望[J]. 信息资源管理学报, 2025, 15(1): 13-20. (XU H, CHENG Q X, DONG J, et al. Generation AI in human-AI interaction: origins, characteristics and future prospects[J]. Journal of information resource management, 2025, 15(1): 13-20.)
[16] 王晰巍, 赵可轶, 刘宇桐, 等. 智能语音交互用户使用影响因素及使用行为组态分析[J]. 图书馆论坛, 2023, 43(12): 147-160. (WANG X W, ZHAO K Y, LIU Y T, et al. Configuration analysis on intelligent voice interaction: influencing factors and user behavior[J]. Library tribune, 2023, 43(12): 147-160.)
[17] HINDERKS A, SCHREPP M, MAYO F J D, et al. Developing a UX KPI based on the user experience questionnaire[J]. Computer standards & interfaces, 2019, 65: 38-44.
[18] CHENG Y, JIANG H. AI-powered mental health chatbots: examining users' motivations, active communicative action and engagement after mass-shooting disasters[J]. Journal of contingencies and crisis management, 2020, 28(3): 339-354.
[19] BATTARBEE K, KOSKINEN I. Co-experience: user experience as interaction[J]. CoDesign, 2005, 1(1): 5-18.
[20] SOUZA K E S, AVIZ I L, MELLO H D, et al. An evaluation framework for user experience using eye tracking, mouse tracking, keyboard input, and artificial intelligence: a case study[J]. International journal of human-computer interaction, 2022, 38(7): 646-660.
[21] SABUKUNZE I D, ARAKAZA A. User experience analysis on mobile application design using user experience questionnaire[J]. Indonesian journal of information systems, 2021, 4(1): 15-26.
[22] 王伟伟, 宁瑨, 魏婷.基于认知负荷的用户感知体验情感评价方法[J]. 包装工程, 2022, 43(4): 147-155. (WANG W W, NING J, WEI T. Emotional evaluation method of user perceptual experience based on the perspective of cognitive load[J]. Packaging engineering, 2022, 43(4): 147-155.)
[23] HART S G, STAVELAND L E. Development of NASA-TLX (task load index): results of empirical and theoretical research[J]. Advances in psychology, 1988, 52:139-183.
[24] RODDEN K, HUTCHINSON H, FU X. Measuring the user experience on a large scale: user-centered metrics for web applications[C]//Proceedings of the SIGCHI conference on human factors in computing systems. New York: SIGCHI, 2010:2395-2398.
[25] 张朋云.基于眼动和皮肤电信号的认知负荷测量研究[D]. 武汉:华中师范大学, 2023. (ZHANG P Y. Cognitive load measurement based on eye tracking and electrical skin signals[D]. Wuhan: Central China Normal University, 2023.)
[26] 李立睿, 张嘉程, 邓小昭.图书馆机器人服务用户体验影响机理研究[J]. 图书情报工作, 2024, 68(12): 43-55. (ZHANG L R, ZHANG J C, DENG X Z. Research on the influencing mechanism of user experience of robot services in libraries[J]. Library and information service, 2024, 68(12): 43-55.)
[27] 穆肃, 崔萌, 黄晓地.全景透视多模态学习分析的数据整合方法[J]. 现代远程教育研究, 2021, 33(1): 26-37, 48. (MU S, CUI M, HUANG X D. Data fusion method in multimodal learning analytics: from a panoramic perspective[J]. Modern distance education research, 2021, 33(1): 26-37, 48.)
[28] 陈其晖, 李诗吟, 杨劲松.虚拟社群用户学习绩效及其影响机制研究——基于自我决定动机理论[J]. 成人教育, 2024, 44(7): 41-49. (CHEN Q H, LI S Y, YANG J S. Research on learning performance of virtual community users and its influencing mechanism: based on the theory of self-determination motivation[J]. Adult education, 2024, 44(7): 41-49.)
[29] 吴丹, 刘春香.交互式信息检索研究中的眼动追踪分析[J]. 中国图书馆学报, 2019, 45(2): 109-128. (WU D, LIU C X. Eye-tracking analysis in interactive information retrieval[J]. Journal of library science in China, 2019, 45(2): 109-128.)
[30] 陈凯泉, 张春雪, 吴玥玥, 等. 教育人工智能(EAI)中的多模态学习分析、适应性反馈及人机协同[J]. 远程教育杂志, 2019, 37(5): 24-34. (CHEN K Q, ZHANG C X, WU Y Y, et al. Multi-modal learning analysis, adaptive feedback and human-computer coordination of artificial intelligence in education(EAI)[J]. Journal of distance education, 2019, 37(5): 24-34.)
[31] 苏文成, 于莹莹, 卢章平, 等. 高校图书馆用户学习与阅读行为的空间嗅觉影响机理研究[J]. 图书馆建设, 2024(2): 81-93. (SU W C, YU Y Y, LU Z P, et al. Study on the mechanism of spatial olfactory influence on learning and reading behavior of academic library users[J]. Library development, 2024(2): 81-93.)
[32] KUO Y C, YAO C B, WU C Y. A strategy for enhancing English learning achievement, based on the eye-tracking technology with self-regulated learning[J]. Sustainability, 2022, 14(23): 16286.
[33] SERRANO-MAMOLAR A, MIGUEL-ALONSO I, CHECA D, et al. Towards learner performance evaluation in iVR learning environments using eye-tracking and machine-learning[J]. Comunicar: media education research journal, 2023, 31(76): 9-19.
[34] JIAN Y C. Eye-movement patterns and reader characteristics of students with good and poor performance when reading scientific text with diagrams[J]. Reading and writing, 2017, 30(7): 1447-1472.
[35] CHETTAOUI N, ATIA A, BOUHLEL M S. Student performance prediction with eye-gaze data in embodied educational context[J]. Education and information technologies, 2023, 28(1): 833-855.
[36] 张家华, 张剑平.学习过程信息加工模型的演变与思考[J]. 电化教育研究, 2011(1): 40-43. (ZHANG J H, ZHANG J P. The evolution and reflections on the information processing model in the learning process[J]. E-education research, 2011(1): 40-43.)
[37] 郭梦瑶, 徐进波, 夏宇诚.多模态交互方式的用户接受度研究[J]. 艺术与设计(理论), 2019, 2(12): 104-106. (GUO M Y, XU J B, XIA Y C. Research on user acceptance of multimodal interaction design[J]. Art and design, 2019, 2(12): 104-106.)
[38] 谭晓艳.学习动机理论在小学数学教学中的应用策略研究[J]. 教师, 2025, (1): 71-73. (TAN X Y. Research on application strategies of learning motivation theories in elementary school mathematics teaching[J]. Teacher, 2025(1): 71-73.)
[39] 周进, 叶俊民, 李超.多模态学习情感计算:动因、框架与建议[J]. 电化教育研究, 2021, 42(7): 26-32, 46. (ZHOU J, YE J M, LI C. Multimodal learning affective computing: motivations, frameworks, and suggestions[J]. E-education research, 2021, 42(7): 26-32, 46.)
[40] 胡翰林, 冯瑞.多模态学习的研究进展与趋势:跨学科的透视[J]. 黑龙江高教研究, 2024, 42(6): 148-154. (HU H L, FENG R. Research progress and trends in multimodal learning: an interdisciplinary perspective[J]. Heilongjiang researches on higher education, 2024, 42(6): 148-154.)
[41] JACOB R J K, KARN K S. Commentary on section 4-eye tracking in human-computer interaction and usability research: ready to deliver the promises[J]. The mind’s eye, 2003, 2(3): 573-605.
[42] 陈静, 张璐.感受性视角下人智交互研究:主题挖掘、方法识别、未来发展[J]. 情报理论与实践, 2024, 47(2): 98-106. (CHEN J, ZHANG L. Human-AI interaction research from a receptivity perspective: topic classification, method identification, and future development[J]. Information studies: theory & application, 2024, 47(2): 98-106.)
[43] 苏文成, 吴君如, 于莹莹, 等. 虚拟现实场景中用户沉浸式信息检索绩效的影响因素研究[J]. 图书情报工作, 2024, 68(7): 22-35. (SU W C, WU J R, YU Y Y, et al. Study on factors influencing users’ immersive information retrieval performance in virtual reality contexts[J]. Library and information service, 2024, 68(7): 22-35.)
[44] 郑兰琴, 高蕾, 黄梓宸.基于生成式人工智能技术的对话机器人能促进在线协作学习绩效吗?[J]. 电化教育研究, 2024, 45(3): 70-76, 84. (ZHENG L Q, GAO L, HUANG Z C. Can chatbots based on generative artificial intelligence facilitate online collaborative learning performance?[J]. E-education research, 2024, 45(3): 70-76, 84.)
[45] 苏文成, 郭浩然, 卢章平, 等. 我国高校学生群体人工智能素养评价指标体系构建及实效性验证[J]. 图书馆建设, 2025(1): 94-105. (SU W C, GUO H R, LU Z P, et al. Construction of artificial intelligence literacy evaluation index system and validation of its effectiveness in China’s college and university student population[J]. Library development, 2025(1): 94-105.)
[46] 李泽. 面向在线学习场景的眼动与皮肤电信号的情感识别研究[D]. 桂林: 桂林理工大学, 2023. (LI Z. Research on emotion recognition of eye movement signals and electrodermal activity signals for online learning scenarios[D]. Guilin: Guilin University of Technology, 2023.)
[47] 陈丽先, 曾妮, 石冰, 等. Asher-McDade鼻唇评价量表的汉化及信效度初步研究[J]. 华西口腔医学杂志, 2024, 42(1): 97-103. (CHEN L X, ZENG N, SHI B, et al. Preliminary study on reliability and validity of the Chinese version of the Asher-Mc Dade aesthetic index[J]. West China journal of stomatology, 2024, 42(1): 97-103.)
[48] 赵恬, 丁妮.提升用户体验的VR眼动追踪系统:技术及应用[J]. 现代电影技术, 2024(2): 43-49. (ZHAO T, DING N. A research towards the technology and application of eye tracking system in VR for enhancing user experience[J]. Advanced motion picture technology, 2024(2): 43-49.).
[49] 陈静, 黎雅丽, 陆泉.基于眼动追踪的用户感知预测模型研究[J]. 情报理论与实践, 2022, 45(4): 154-161, 169. (CHEN J, LI Y L, LU Q. Research on user perception prediction model based on eye tracking[J]. Information studies: theory & application, 2022, 45(4): 154-161, 169.)
[50] ZAGERMANN J, PFEIL U, REITERER H. Measuring cognitive load using eye tracking technology in visual computing[C]//Proceedings of the sixth workshop on beyond time and errors on novel evaluation methods for visualization. New York: BELIV, 2016:78-85.
[51] 王晰巍, 刘宇桐, 乌吉斯古楞, 等. AI语音助手用户虚拟在线体验影响因素[J]. 图书馆论坛, 2024, 44(1): 71-85. (WANG X W, LIU Y T, WUJI S G L, et al. Research on the influencing factors of AI voice assistant user’s virtual online experience[J]. Library tribune, 2024, 44(1): 71-85.)
[52] 来元.大学英语的多模态教学与学生的学习绩效研究[J]. 淮海工学院学报(人文社会科学版), 2013, 11(15): 63-66. (LAI Y. A study of multimodal college English teaching and learning efficiency[J]. Journal of Jiangsu Ocean University(humanities & social sciences edition), 2013, 11(15): 63-66.)
[53] HENDERSON J M, FERREIRA F. Effects of foveal processing difficulty on the perceptual span in reading: implications for attention and eye movement control[J]. Journal of experimental psychology: learning, memory, and cognition, 1990(3): 417-429.
[54] PéREZ-MARíN D, PAREDES-VELASCO M, PIZARRO C. Multi-mode digital teaching and learning of human-computer interaction (HCI) using the VARK model during COVID-19[J]. Educational technology & society, 2022, 25(1): 78-91.
[55] 郭亚军, 庞义伟, 周家华, 等. ChatGPT赋能图书馆虚拟数字人:技术优势、应用场景与实践路径[J]. 图书馆论坛, 2024, 44(8): 69-79. (GUO Y J, PANG Y W, ZHOU J H, et al. ChatGPT empowers digital humans in libraries: technological advantages, application scenarios, and practical pathways[J]. Library tribune, 2024, 44(8): 69-79.)
[56] 肖自力.智慧图书馆中基于人工智能的个性化阅读推荐系统设计实施策略[J]. 兰台内外, 2024(10): 70-72. (XIAO Z L. Design and implementation strategies of AI-based personalized reading recommendation system in smart libraries[J]. Inside and outside lantai, 2024(10): 70-72.)
[57] 张鹏.生成式人工智能聊天机器人:重新定义图书馆用户服务[J]. 图书馆理论与实践, 2024(2): 123-129. (ZHANG P. Generative artificial intelligence chatbots: redefining user service in library[J]. Library theory and practice, 2024(2): 123-129.)
[58] 李颖婷.生成式人工智能给图书馆带来的机遇、挑战及应对策略[J]. 图书与情报, 2023(2): 42-48. (LI Y T. Opportunities, challenges, and strategies for libraries in the era of artificial intelligence generated content[J]. Library & information, 2023(2): 42-48.)
[59] 黄如花, 石乐怡, 江语蒙.6I未来学习中心:高校图书馆数智赋能教学科研服务新探[J]. 中国图书馆学报, 2024, 50(5): 42-58. (HUANG R H, SHI L Y, JIANG Y M. 6I learning center for the future: digital intelligence empowerment for teaching and research services in university libraries[J]. Journal of library science in China, 2024, 50(5): 42-58.)
[60] 姚跃林, 胡小蓉, 林雯萍, 等. 双减背景下人工智能服务课后学习的实践与研究[J]. 中国教育技术装备, 2024(5): 94-100. (YAO Y L, HU X R, LIN W P, et al. Practice and research on artificial intelligence services for after-school learning under the background of double reduction[J]. China educational technology & equipment, 2024(5): 94-100.)
[61] 柯和平, 龚自振, 谢海先.智慧图书馆建设中混合式智慧学习空间的构建研究[J]. 现代教育技术, 2024, 34(4): 112-121. (KE H P, GONG Z Z, XIE H X. Research on the construction of mixed smart learning space of smart library[J]. Modern educational technology, 2024, 34(4): 112-121.)
Outlines

/

〈 〉