[Purpose/Significance] In recent years, there has been a trend toward digital collections, that are multisensory, realistic, and interactive. New resource forms such as 3D collections and panoramic videos have become important forms of recording and disseminating knowledge. It is difficult to support these situation-based interactive digital cultural services of public institutions by traditional user interface, such as libraries, museums and archives. Therefore, public needs and expanding interactive situations call for the new user interface with immersive and natural characteristics. [Method/Process] Based on the discussion of the related issues on digital collection resources and services, multi-mode interface and experience, this paper proposed the multi-modal interface model for digital collections. Based on the information processing model of multi-modal interaction, the information exchange process of multi-mode interaction should be constrained by people’s information processing ability. By giving a multi-mode interactive information processing model, the interactive cognition process of collection resources in virtual reality is analyzed. Then, based on the multi-mode interaction model for collection resources, according to the progressive logic of “interface-cognition-context”, this study analyzed the key factors affecting user experience (UE). [Result/Conclusion] The multi-mode interaction interface for digital collections is composed of 3 progressive levels of interaction modality, interaction primitive and interaction task. Human-centered multi-mode interaction could enhance human’s cognitive abilities in the stages of multi-mode perception, thinking decision-making and interactive execution. The key influencing factors of immersive UE involve perceptual and interactive modality, matching degree of mental model, immersion dimension of interaction scenario, which can provide reference for multimode interaction design as well as digital culture services.
[1] 肖希明, 唐义. 图书馆学博物馆学档案学课程体系整合初探[J]. 中国图书馆学报, 2014, 40(3):4-12.
[2] 夏立新, 白阳, 张心怡. 融合与重构:智慧图书馆发展新形态[J]. 中国图书馆学报, 2018, 44(1):35-49.
[3] FLORENCE W. The best online culture archives[EB/OL].[2022- 08-30]. https://www.telegraph.co.uk/culture/8296365/The-bestonline-culture-archives.html.
[4] 曾蕾, 王晓光, 范炜. 图档博领域的智慧数据及其在数字人文研究中的角色[J]. 中国图书馆学报, 2018, 44(1):17-34.
[5] 卢小宾, 宋姬芳, 蒋玲, 等. 智慧图书馆建设标准探析[J]. 中国图书馆学报, 2021, 47(1):15-33.
[6] 张晓林, 许旭. 让数字图书馆驱动图书馆服务创新发展- 读《国际图联数字图书馆宣言》有感[J]. 中国图书馆学报, 2010, 36(3):73-76.
[7] 朱学芳. 图博档信息资源数字化建设及服务融合探讨[J]. 情报资料工作, 2011(5):57-60.
[8] CARVAJAL D A L, MORITA M M, BILMES G M. Virtual museums. captured reality and 3D modeling[J]. Journal of cultural heritage, 2020, 45:234-239.
[9] HANNAH M, HUBER S, MATEI S A. Collecting virtual and augmented reality in the twenty-first century library[J]. Collection management, 2019, 44(2/4):277-295.
[10] FENU C, PITTARELLO F. Svevo tour:the design and the experimentation of an augmented reality application for engaging visitors of a literary museum[J]. International journal of humancomputer studies, 2018, 114:20-35.
[11] 张宁, MIGUEL B N, 李俊炀, 等. 面向中华古籍阅读推广与文化传播的VR系统模型构建与实现[J]. 图书情报工作, 2021, 65(13):12-24.
[12] PETRIDIS P, PLETINCKX D, MANIA K, et al. The EPOCH multimodal interface for interacting with digital heritage artefacts[C]//ZHA H, PAN Z, THWAITES H, et al. International conference on virtual systems and multimedia. Berlin:Springer, 2006:408-417.
[13] CHI T V, ABLART D, GATTI E, et al. Not just seeing, but also feeling art:mid-air haptic experiences integrated in a multisensory art exhibition[J]. International journal of humancomputer studies, 2017, 108:1-14.
[14] WÓJCIK M. Haptic technology-potential for library services[J]. Library hi tech, 2019, 37(17):883-893.
[15] REITERER H, RAEDLE R, MUELLER J. Blended library:new ways to access the collections of scientific and public libraries[J]. Bibliothek forschung und praxis, 2016, 40(1):7-20.
[16] PARK C H, RYU E S, HOWARD A M. Telerobotic haptic exploration in art galleries and museums for individuals with visual impairments[J]. IEEE transactions on haptics, 2015, 8(3):327-338.
[17] CECILIA R R. COVID-19 Pandemic:threat or opportunity for blind and partially sighted museum visitors?[J]. Journal of conservation and museum studies, 2021, 19(1):article 5.
[18] CONFALONIERI R, YEE-KING M, HAZELDEN K, et al. Engineering multiuser museum interactives for shared cultural experiences[J]. Engineering applications of artificial intelligence, 2015, 46:180-195.
[19] XU W, DAI T T, SHEN Z Y, et al. Effects of technology application on museum learning:a meta-analysis of 42 studies published between 2011 and 2021[J]. Interactive learning environments, 2021:1-16.
[20] 祁彬斌, 项欣, 朱学芳, 等. 自然人机交互在图博档中的应用分析[J]. 图书情报知识, 2019(3):53-61.
[21] MCTEAR M F. Spoken dialogue technology:enabling the conversational user interface[J]. ACM computing surveys (CSUR), 2002, 34(1):90-169.
[22] LUZHNICA G, SIMON J, LEX E, et al. A sliding window approach to natural hand gesture recognition using a custom data glove[C]//2016 IEEE symposium on 3D user interfaces (3DUI). Twoson:IEEE, 2016:81-90.
[23] AGGARWAL R, VOHRA A, NAMBOODIRI A M. Panoramic stereo videos with a single camera[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. Las Vegas:IEEE Computer Society, 2016:3755-3763.
[24] CHOW S K, CHAN K L. Reconstruction of photorealistic 3D model of ceramic artefacts for interactive virtual exhibition[J]. Journal of cultural heritage, 2009, 10(2):161-173.
[25] CLERCKX T, VANDERVELPEN C, CONINX K. Task-based design and runtime support for multimodal user interface distribution[C]//GULLIKSEN J, HARNING M B, PALANQUE P, et al. IFIP international conference on engineering for humancomputer interaction. Berlin:Springer, 2007:89-105.
[26] 张凤军, 戴国忠, 彭晓兰. 虚拟现实的人机交互综述[J]. 中国科学:信息科学, 2016, 46(12):1711-1736.
[27] 田丰, 戴国忠, 陈由迪, 等. 三维交互任务的描述和结构设计[J]. 软件学报, 2002(11):2099-2105.
[28] 张小龙, 吕菲, 程时伟. 智能时代的人机交互范式[J]. 中国科学:信息科学, 2018(4):406-418.
[29] 祁彬斌, 朱学芳. 引入力触觉的数字文物多模交互方法[J]. 中国图象图形学报, 2018, 23(8):1218-1230.
[30] 祁彬斌, 胡玉宁, 朱学芳, 等. 面向盲人阅读的触觉交互服务框架构建与应用研究[J]. 图书情报工作, 2019, 63(14):20-29.
[31] WICKENS C D, HOLLANDS J G, PARASURAMAN R, et al. Engineering psychology & Human performance[M]. 4th ed. Boston:Pearson, 2012.
[32] MAYER R E. Multimedia learning[M]. Cambridge:Cambridge University Press, 2001.
[33] 饶培伦. VR交互的人因研究[J]. 中国计算机学会通讯, 2018, 14(5):20-25.
[34] ZHANG L, DUAN H, QIN S, et al. High cortisol awakening response is associated with impaired error monitoring and decreased post-error adjustment[J]. Stress, 2015, 18(5):561-568.
[35] ZHANG L, QIN S, YAO Z, et al. Long-term academic stress enhances early processing of facial expressions[J]. International journal of psychophysiology, 2016, 109:138-146.
[36] ERNST M O, BANKS M S. Humans integrate visual and haptic information in a statistically optimal fashion[J]. Nature, 2002, 415(6870):429-433.
[37] 王党校, 郑一磊, 李腾, 等. 面向人类智能增强的多模态人机交互[J]. 中国科学:信息科学, 2018, 48(4):95-111.
[38] HAMZA-LUP F G, STANESCU I A. The haptic paradigm in education:challenges and case studies[J]. Internet & higher education, 2010, 13(1/2):78-81.
[39] 项欣, 祁彬斌, 朱学芳. 图博档馆藏实体多模交互式呈现的可用性评估[J]. 图书馆论坛, 2019, 39(1):92-99.
[40] KORENBERG. Mental modelling[J]. Nature genetics, 11(2):109-111.
[41] 刘金平, 周广亚, 刘亚丽. 情境启动和认知需要对决策中信息加工的影响[J]. 心理科学, 2008(2):315-318, 303.
[42] MAKRANSKY G, PETERSEN G B. The cognitive affective model of immersive learning (CAMIL):a theoretical researchbased model of learning in immersive virtual reality[J]. Educational psychology review, 2021, 33(3):937-958.
