Research on Users' Emotions and Impact Factors Based on Facial Expression Recognition in Exploratory Search

Huang Kun; Zheng Mingxuan; Luo Shichao; Jin Jian

doi:10.13266/j.issn.0252-3116.2022.05.010

Library and Information Service >

2022 , Vol. 66 >Issue 5: 93 - 104

DOI: https://doi.org/10.13266/j.issn.0252-3116.2022.05.010

INFORMATION RESEARCH

Research on Users' Emotions and Impact Factors Based on Facial Expression Recognition in Exploratory Search

Huang Kun ,
Zheng Mingxuan ,
Luo Shichao ,
Jin Jian

Expand

School of Government, Beijing Normal University, Beijing 100875

Received date: 2021-07-22

Revised date: 2021-11-27

Online published: 2022-03-21

Fold

Abstract

[Purpose/significance] Combined with facial expression recognition technology, this study focuses on the relationship between emotions and related factors in the search process, and explores the relationship among emotions, search interaction behaviors, as well as user experience in the search process, so as to provide references for automatic recognition and prediction of users’ emotional state based on search interaction behaviors, and serve for the optimization of the search interaction process. [Method/process] Through experimental research, 48 subjects were recruited and divided into two groups. They completed three designated tasks with or without time constraints respectively, and their expression data, behavior data and related self-evaluation data were collected. [Result/conclusion] The results show that: in the search process, users’ neutral emotions account for the highest proportion (58.03%), followed by negative emotions (29.88%), and positive emotions account for the least proportion (12.10%). The higher the proportion of users’ non-neutral emotions, the worse the post-search experience, and conversely, the higher the proportion of neutral emotions, the better the users’ post-search experience. The proportion of sadness, disgust, surprise and happiness in the time limited group is significantly higher than that in the non-time limited group; at the same time, the proportion of neutral emotions in the time limited group is significantly lower than that of the non-time limit group. The high task difficulty perception group shows more disgust, anger and less neutral emotions. Non-neutral emotions are more likely to occur in page switching scenarios. The more neutral emotions in the search process, the better the user’s experience after the search, while the proportion of negative emotions and positive emotions is negatively correlated with the experience after the search.

Key words： emotion; facial expression recognition; exploratory search; impact factor

Cite this article

Huang Kun , Zheng Mingxuan , Luo Shichao , Jin Jian . Research on Users' Emotions and Impact Factors Based on Facial Expression Recognition in Exploratory Search[J]. Library and Information Service, 2022 , 66(5) : 93 -104 . DOI: 10.13266/j.issn.0252-3116.2022.05.010

References

[1] NAHL D. Measuring the affective information environment of Web searchers[J].Proceedings of the Association for Information Science and Technology,2004,41(1):191-197.
[2] MARCHIONINI G. Exploratory search:from finding to understanding[J].Communications of the ACM,2006,49(4):41-46.
[3] WHITE R W, ROTH R A. Exploratory search:beyond the query-response paradigm[J]. Synthesis lectures on information concepts retrieval and services,2009, 1(1): 1-98.
[4] SCOTT B,CLIFFORD N,KEVIN H. Computers that care:investigating the effects of orientation of emotion exhibited by an embodied computer agent[J]. International journal of human-computer studies, 2005, 62(2):161-178.
[5] 李京津,黄崑,袁心,等.信息查询与检索领域中用户情感因素的界定与应用[J].图书情报工作,2019,63(7):63-71.
[6] 郭婷婷,崔丽霞,王岩.情绪复杂性:探讨情绪功能的新视角[J].心理科学进展,2011,19(7):993-1002.
[7] ARAPAKIS I, JOSE J M, GRAY P D. Affective feedback:an investigation into the role of emotions in the information seeking process[C]// Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval.Singapore:ACM, 2008:395-402.
[8] ARAPAKIS I, MOSHFEGHI Y, JOHO H, et al. Enriching user profiling with affective features for the improvement of a multimodal recommender system[C]// ACM international conference on image & video retrieval. Santorini:ACM, 2009:1-8.
[9] ARAPAKIS I, KONSTAS I, JOSE J M, et al. Modeling facial expressions and peripheral physiological signals to predict topical relevance[C]//Proceedings of the 32nd international ACM SIGIR conference on research and development in information retrieval. Boston:ACM,2009:728-729.
[10] LOPATOVSKA I. Emotional aspects of the online information retrieval process [D]. New Brunswick:Rutgers The State University of New Jersey, 2009.
[11] LOPATOVSKA I. Emotional correlates of information retrieval behaviors[C]// Affective computational intelligence. Paris:IEEE, 2011:1-7.
[12] LOPATOVSKA I. Toward a model of emotions and mood in the online information search process[J]. Journal of the Association for Information Science & Technology, 2014, 65(9):1775-1793.
[13] ZANGANEH M Y, HARIRI N. The role of emotional aspects in the information retrieval from the Web[J]. Online information review, 2018, 42(4):520-534.
[14] LOPATOVSKA I. Searching for good mood:examining relationships between search task and mood[J]. Proceedings of the American Society for Information Science & Technology, 2009, 46(1):1-13.
[15] KOO J H. Information seeking within negative affect[J]. Journal of the Korean Society for Library and Information Science, 2016, 50(1):285-312.
[16] 周文杰. 数字信息分析中用户焦虑实验研究[J]. 中国图书馆学报, 2011, 37(6):58-66.
[17] KRACKER J. Research anxiety and students' perceptions of research:an experiment. part I. effect of teaching Kuhlthau's ISP model[J]. Journal of the Association for Information Science and Technology, 2002, 53(4):282-294.
[18] KRACKER J, WANG P. Research anxiety and students' perceptions of research:an experiment. part II. content analysis of their writings on two experiences[J]. Journal of the Association for Information Science and Technology, 2002, 53(4):295-307.
[19] 姜婷婷,贺虹虹,张正楠. 搜索任务复杂度对用户情感的影响研究[J].图书情报知识,2016(4):74-82.
[20] ONWUEGBUZIE A J, JIAO Q G, BOSTICK S L. Library anxiety:theory, research, and applications[J]. Library review, 2004, 56(3):258-259.
[21] NAHL D. Affective and cognitive information behavior:interaction effects in Internet use[J]. Proceedings of the Association for Information Science and Technology, 2005, 42(1):246-268.
[22] LAZAR J, JONES A, HACKLEY M, et al. Severity and impact of computer user frustration:a comparison of student and workplace users[J]. Interacting with computers, 2006, 18(2):187-207.
[23] TENOPIR C, WANG P, ZHANG Y, et al. Academic users' interactions with ScienceDirect in search tasks:affective and cognitive behaviors[J]. Information processing & management, 2008, 44(1):105-121.
[24] SHANNON D. Kuhlthau's information search process[J]. School library media activities monthly, 2002, 19(2):19-19.
[25] 邱瑾,吴丹.协同信息检索行为中的情感研究[J].图书与情报, 2013(2):105-110.
[26] 袁红,赵宇珺.协同搜索行为中的用户任务感知及情绪状态研究[J].图书情报工作,2015, 59(17):89-98.
[27] CRESCENZI A, CAPRA R, ARGUELLO J. Time pressure, user satisfaction and task difficulty[J]. Proceedings of the American Society for Information Science and Technology, 2013, 50(1):1-4.
[28] IMAZU M, NAKAYAMA S, JOHO H. Effect of explicit roles on collaborative search in travel planning task[C]// Asia Information Retrieval Symposium. Dubai:Springer,2011:205-214.
[29] 黄崑,袁心,李蕾,等.情感负荷视角下协作检索中用户消极情绪感知及其影响因素研究[J].图书情报知识,2020(1):42-52.
[30] 王昌,韩建华,孙晓宁.协同信息搜寻用户情绪影响因素研究--基于自我效能感和任务复杂性[J].图书情报知识,2021(1):76-84.
[31] GWIZDKA J, LOPATOVSKA I. The role of subjective factors in the information search process[J]. Journal of the Association for Information Science & Technology, 2014, 60(13):2452-2464.
[32] 谭金波,王广新.学习情绪与查询任务类型的交互作用对网络信息搜索体验的影响[J].中国远程教育,2017(9):19-25,79.
[33] ANDERSEN R A, NASROLLAHI K, MOESLUND T B, et al. Interfacing assessment using facial expression recognition[D].Lisbon:University of Warwick, 2014.
[34] GHIJSEN M, NIJHOLT I A, POEL M, et al. Facial expression analysis for human computer interaction-recognizing emotions in an intelligent[J]. Affective computing, 2004, 2(3):147-161.
[35] 胡敏,綦科. 表情识别技术的教育应用[J].教育现代化,2019,6(75):100-102.
[36] PASCHERO M, VESCOVO G D, BENUCCI L, et al. A real time classifier for emotion and stress recognition in a vehicle driver [C]// IEEE international symposium on industrial electronics. Hangzhou:IEEE, 2012:1690-1695.
[37] YANG Z, KAY A, LI Y, et al. Pose-based body language recognition for emotion and psychiatric symptom interpretation[C]// 2020 25th international conference on pattern recognition. Electr Network:IEEE,2021:294-301.
[38] EKMAN P, FRIESEN W V. Facial action coding system:a technique for the measurement of facial movement [M].Palo Alto:Consulting Psychologists Press, 1978.
[39] ATHUKORALA K, DOROTA GŁOWACKA, JACUCCI G, et al. Is exploratory search different? a comparison of information search behavior for exploratory and lookup tasks[J]. Journal of the Association for Information Science and Technology, 2016, 67(11):2635-2651.
[40] ANDERSON L W, KRATHWOHL D R, AIRASIAN P W, et al. A taxonomy for learning, teaching, and assessing:a revision of bloom’s taxonomy of educational objectives[M]. New York:Longman,2001.
[41] KELLY D,ARGUELLO J,EDWARDS A, et al. Development and evaluation of search tasks for IIR experiments using a cognitive complexity framework[C]//Proceedings of the 2015 international conference on the theory of information retrieval. Northampton:ACM,2015:101-110.
[42] 李京津,黄崑,袁心,等.信息查询与检索领域中用户情感因素的界定与应用[J].图书情报工作,2019,63(7):63-71.
[43] SCHERER K R. What are emotions? and how can they be measured?[J].Social science information, 2005, 44(4) :695-729.
[44] CHEN H, GU Y, WANG F, et al. Facial expression recognition and positive emotion incentive system for human-robot interaction[C]// Proceedings of the 13th world congress on intelligent control and automation. Changsha:IEEE, 2019:407-412.
[45] 李浩然.旷视人工智能开放平台Face++:长尾图像应用的支撑[J].人工智能,2019(2):76-86.
[46] EKMAN P, FRIESEN W V. Unmasking the face[M]. Cambridge MA:Malor Books, 2003.
[47] 李珊,邓伟洪.深度人脸表情识别研究进展[J].中国图象图形学报,2020,25(11):2306-2320.
[48] WATSON D, CLARK L A, TELLEGEN A. Development and validation of brief measures of positive and negative affect:the PANAS scales.[J]. Journal of personality and social psychology,1988,54(6):1063-1070.
[49] WATSON D, WIESE D, VAIDYA J, et al. The two general activation systems of affect:structural findings, evolutionary considerations, and psychobiological evidence[J]. Journal of personality and social psychology, 1999, 76(5):820-838.
[50] 邱林,郑雪,王雁飞.积极情感消极情感量表(PANAS)的修订[J].应用心理学,2008,14(3):249-254,268.
[51] ARGUELLO J. Predicting search task difficulty[C]//European conference on information retrieval. Amsterdam:Springer, 2014:88-99.
[52] LIU C, LIU J, BELKIN N J. Predicting search task difficulty at different search stages[C]//Proceedings of the 23rd ACM international conference on information and knowledge management. Shanghai: ACM, 2014:519-578.
[53] LIU J, LIU C, COLE M, et al. Exploring and predicting search task difficulty[C]//Proceedings of the 21st ACM international conference on information and knowledge management. Maui:ACM, 2012:1313-1322.
[54] 艾卉,周慧,黄宇霞. 情绪的跨文化普遍性与差异性——对教育实践的启示[J].全球教育展望,2015,44(11):104-113.
[55] LIM N. Cultural differences in emotion:differences in emotional arousal level between the East and the West-ScienceDirect[J]. Integrative medicine research, 2016, 5(2):105-109.
[56] SHARMA P, JOSHI S, GAUTAM S, et al. Student engagement detection using emotion analysis, eye tracking and head movement with machine learning [EB/OL].[2022-01-21]. https://arxiv.org/abs/1909.12913.
[57] 魏艳涛,雷芬,胡美佳,等.学生表情识别研究综述[J].中国教育信息化,2020(21):48-55.
[58] DMELLO S K, GRAESSER A C, PICARD R W. Toward an affect-sensitive autoTtutor[J]. Intelligent systems, IEEE, 2007, 22(4):53-61.
[59] MCDANIEL B, DMELLO S, KING B, et al. Facial features for affective state detection in learning environments[J]. Cognitive science society, 2007:467-472.
[60] 冯莹,甘怡群,柳之啸,等.不确定性反应风格和未来取向应对的关系探究:预期情绪的中介作用[J].北京大学学报(自然科学版),2015,51(3):485-494.
[61] VERONICA G,DEREK R. Coping with uncertainty:the construction and validation of a new measure[J]. Personality and individual differences, 2001, 31(4):519-534.
[62] 刘畅,赵瑜,杨帆.信息检索用户实验设计中时间限制和任务次序的影响研究[J].图书情报工作,2015,59(1):99-105.
[63] MAULE A J, HOCKEY G, BDZOLA L. Effects of time-pressure on decision-making under uncertainty:changes in affective state and information processing strategy[J]. Acta psychologica, 2000, 104(3):283-301.
[64] CHEN S Y, RIEH S Y. Take your time first, time your search later:how college students perceive time in Web searching[J]. Proceedings of the American Society for Information Science and Technology, 2009, 46(1):1-19.
[65] NAHL D. Measuring the affective information environment of Web searchers[C]//Proceedings of the 67th annual meeting of the American Society for Information Science & Technology.Medford:Information Today,2004:191-197.
[66] 徐振国,张冠文,孟祥增,等. 基于深度学习的学习者情感识别与应用[J].电化教育研究,2019,40(2):87-94.
[67] WEHNER C, SCHILS T, BORGHANS L. Personality and mental health:the role and substitution effect of emotional stability and conscientiousness [EB/OL].[2022-01-21]. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2864838
[68] QURTUBI A. Analysis of students’ emotional stability, parenting style and teaching method on student learning achievement :a case study of junior high school student in jakarta[J]. Asian journal of management sciences & education, 2016, 5(2):101-109.
[69] KUMARAVELU G. Emotional stability of high school students in relation to their selected variables[J]. Journal of emerging technologies and innovative research, 2018, 5(1):167-169.
[70] 邱明辉, 邱雨奇. 支持探索式查询的帮助功能设计研究[J]. 农业图书情报学报, 2021, 33(3):56-67.
[71] 邹薇. 基于SOR模型的高校图书馆员行为对读者参与的影响实证研究[J].农业图书情报,2019,31(10):76-81.
[72] WHITEHILL J, SERPELL Z, LIN Y C, et al. The faces of engagement:automatic recognition of student engagement from facial expressions[J]. IEEE transactions on affective computing, 2014, 5(1):86-98.
[73] WALLBOTT H G. Bodily expression of emotion[J]. European journal of social psychology, 1998,28(6):879-896.
[74] 陈子健,朱晓亮.基于面部表情的学习者情绪自动识别研究——适切性、现状、现存问题和提升路径[J].远程教育杂志,2019,37(4):64-72.

Options

Outlines

模态框（Modal）标题

Abstract

Cite this article

References