图书情报工作

图书情报工作 >

2022 , Vol. 66 >Issue 9: 128 - 140

DOI: https://doi.org/10.13266/j.issn.0252-3116.2022.09.013

知识组织

文本信息检索中相关性判断的次序效应及其情境因素研究

  • 王志红 ,
  • 曹树金
展开
  • 1. 清华大学计算机科学与技术系, 北京 100084;
    2. 北京信息科学与技术国家研究中心, 北京 100084;
    3. 中山大学信息管理学院, 广州 510006
王志红,博士后

收稿日期: 2021-11-23

  修回日期: 2022-02-16

  网络出版日期: 2022-05-12

基金资助

本文系国家社会科学基金重大项目"基于特定领域的网络资源知识组织与导航机制研究"(项目编号:12&ZD222)和中国博士后面上资助项目"复杂任务下基于用户相关性反馈的会话搜索优化研究"(项目编号:2021M691823)研究成果之一。

收起

Research on Order Effect in Relevance Judgment and Its Contextual Factors in Textual Information Retrieval

  • Wang Zhihong ,
  • Cao Shujin
Expand
  • 1. Department of Computer Science and Technology, Tsinghua University, Beijing 100084;
    2. Beijing National Research Center for Information Science and Technology, Beijing 100084;
    3. School of Information Management, Sun Yat-sen University, Guangzhou 510006

Received date: 2021-11-23

  Revised date: 2022-02-16

  Online published: 2022-05-12

Fold

摘要

[目的/意义]基于相关性分值对搜索结果进行列表式线性排列是目前大多数信息检索系统采用的主要方式。已有研究虽然从理论上批判了这种方式的局限性，指出先后检验和阅读的文献之间是相互影响的，但是并未有实证研究进一步调查相关性判断次序效应发生的情境因素。[方法/过程]针对这一问题，采用2X2X2被试间实验法，招募80名被试，要求每人完成4个主题的实验任务，在实验任务完成前后通过问卷调查等方法收集数据，并采用卡方检验、全概率公式背反以及QQ等式等多种方法进行数据分析，以探究相关性判断的次序效应以及文献间关系、文献呈现形式作为情境因素所产生的影响。[结果/结论]研究结果表明，文献呈现次序会影响相关性判断，并且发现这种效应会根据文献间关系以及文献呈现形式的不同而不同。具体而言，研究发现相对于具有互补关系的文献，具有互斥关系的文献的呈现次序更容易影响相关性判断;另外，相对于文献全文，文献片段的呈现次序更容易影响相关性判断。本研究虽然存在一定的局限性，但是研究结论对于文本信息检索的理论、实践以及研究方法均具有一定的启示。

关键词： 相关性判断; 次序效应; 文献间关系; 文献呈现形式; 量子认知

本文引用格式

王志红 , 曹树金 . 文本信息检索中相关性判断的次序效应及其情境因素研究[J]. 图书情报工作, 2022 , 66(9) : 128 -140 . DOI: 10.13266/j.issn.0252-3116.2022.09.013

Abstract

[Purpose/Significance] The main approach adopted in most information retrieval systems currently is to present the search results list linearly based on the relevance score. Although existing researches have theoretically criticized the limitations of this approach, proposing that there is mutual influence between the document previously examined and read, there is no empirical study to further investigate the contextual factors affecting the order effect of relevance judgment. [Method/Process] In response to this question, this paper adopted the 2X2X2 inter-subject experiment method by recruiting 80 subjects. Each subject was required to complete 4 experimental tasks with different topics. Data was collected by questionnaires before and after each experimental task was completed. In order to explore the order effect of relevance judgment and the influence of inter-document relationships as well as the document presentation types as contextual factors, the collected data was analyzed using various methods, such as chi-square test, law of total probability violation, QQ equation, etc. [Result/Conclusion] The research results confirm that the judgment of relevance can be affected by the presentation order of the documents and this effect is different according to the relationships between documents and document presentation types. Specifically, compared to documents with complementary relationships, the order of documents with contradictory relationship is more likely to influence the relevance judgment. In addition, compared with the full text, the order of the presentation of literature snippets is more likely to affect the relevance judgment. Although this paper has its limitations, the research conclusions obtained have certain theoretical, practical and methodological enlightenment.

Key words： relevance judgment; order effect; inter-document relationship; document presentation type; quantum cognition

参考文献

[1] SARACEVIC T. Relevance:a review of the literature and a framework for thinking on the notion in information science. Part II:Nature and manifestations of relevance[J]. Journal of the American Society for Information Science and Technology, 2007, 58(13):1915-1933.
[2] EISENBERG M, BARRY C. Order effects:a study of the possible influence of presentation order on user judgments of document relevance[J]. Journal of the American Society for Information Science, 1988, 39(5):293-300.
[3] PANG L, AI Q, XU J. Beyond probability ranking principle:modeling the dependencies among documents[C]//Proceedings of the 14th ACM international conference on Web search and data mining. New York:ACM,2021:1137-1140.
[4] TIAMIYU M A, AJIFERUKE I Y. A total relevance and document interaction effects model for the evaluation of information retrieval processes[J]. Information processing & management, 1988, 24(4):391-404.
[5] 王思琦,郭金云.公共服务满意度测量的问题顺序效应:来自一项嵌入性调查实验的证据[J].公共管理评论,2020,2(1):92-115.
[6] TOURANGEAU R, RASINSKI K A. Cognitive processes underlying context effects in attitude measurement[J]. Psychological bulletin, 1988, 103(3):299.
[7] 孙玉, 卢淳, 司继伟,等. 任务呈现方式、任务呈现顺序影响青少年算术策略运用的年龄差异[J]. 心理研究, 2017, 10(1):39-49.
[8] 胥兴安,王立磊,高峰强.旅游广告与网络负面口碑对目的地形象的影响——次序效应和交互效应的实验检验[J].旅游学刊,2017,32(12):37-48.
[9] ERT E, FLEISCHER A. Mere position effect in booking hotels online[J]. Journal of travel research, 2016, 55(3):311-321.
[10] 霍航丰. 项目呈现顺序对学习判断的影响[D]. 杭州:浙江师范大学,2017.
[11] PURNAWIRAWAN N, DE PELSMACKER P, DENS N. Balance and sequence in online reviews:how perceived usefulness affects attitudes and intentions[J]. Journal of interactive marketing, 2012, 26(4):244-255.
[12] 谢梦娇. 追加式评论中顺序效应与消费者购买意愿的关系研究[D]. 北京:北京邮电大学, 2018.
[13] 许智慧. 不同难度教学视频中陈述性知识和程序性知识的呈现顺序对学习的影响[D]. 武汉:华中师范大学, 2019.
[14] 王家慰, 邓泳珊, 邢强. 学习顺序和材料相似性对听觉类别学习的影响[J]. 心理与行为研究, 2019, 17(6):750-755.
[15] BERGUS G R, LEVIN I P, ELSTEIN A S. Presenting risks and benefits to patients:the effect of information order on decision making[J]. Journal of general internal medicine, 2002, 17(8):612-617.
[16] 杨文婷. 信息呈现对个体投资决策的影响[D]. 杭州:浙江大学, 2019.
[17] WANG Z, SOLLOWAY T, SHIFFRIN R M, et al. Context effects produced by question orders reveal quantum nature of human judgments[J]. Proceedings of the National Academy of Sciences, 2014, 111(26):9431-9436.
[18] CHU H. Factors affecting relevance judgment:a report from TREC legal track[J]. Journal of documentation, 2011, 67(2):264-278.
[19] XU Y, WANG D. Order effect in relevance judgment[J]. Journal of the American Society for Information Science and Technology, 2008, 59(8):1264-1275.
[20] GOLDMAN E. Search engine bias and the demise of search engine utopianism[M]//Web Search. Berlin:Springer, 2008:121-133.
[21] HUANG M, WANG H. The influence of document presentation order and number of documents judged on users' judgments of relevance[J]. Journal of the American Society for Information Science and Technology, 2004, 55(11):970-979.
[22] DAMESSIE T T, SCHOLER F, JÄRVELIN K, et al. The effect of document order and topic difficulty on assessor agreement[C]//Proceedings of the 2016 ACM international conference on the theory of information retrieval. New York:ACM, 2016:73-76.
[23] DAMESSIE T T, CULPEPPER J S, KIM J, et al. Presentation ordering effects on assessor agreement[C]//Proceedings of the 27th ACM international conference on information and knowledge management. New York:ACM, 2018:723-732.
[24] BRUZA P, CHANG V. Perceptions of document relevance[J]. Frontiers in psychology, 2014, 5(612):1-8.
[25] ZUCCON G, AZZOPARDI L. Using the quantum probability ranking principle to rank interdependent documents[C]//European conference on information retrieval. Berlin:Springer, 2010:357-369.
[26] 李竞飞. 面向探索式信息检索的量子交互模型[D]. 天津:天津大学, 2017.
[27] MA Q, NADAMOTO A, TANAKA K. Complementary information retrieval for cross-media news content[J]. Information systems, 2006, 31(7):659-678.
[28] SCHAMBER L, EISENBERG M B, NILAN M S. A re-examination of relevance:toward a dynamic, situational definition[J]. Information processing & management, 1990, 26(6):755-776.
[29] SCHAMBER L. Relevance and information behavior[J]. Annual review of information science and technology, 1994, 29(1):3-48.
[30] UPRETY S, SONG D. Investigating order effects in multidimensional relevance judgment using query logs[C]//Proceedings of the 2018 ACM SIGIR international conference on theory of information retrieval. New York:ACM,2018:191-194.
[31] LI J, ZHANG P, SONG D, et al. Understanding an enriched multidimensional user relevance model by analyzing query logs[J]. Journal of the Association for Information Science and Technology, 2017, 68(12):2743-2754.
[32] DAMESSIE T T, SCHOLER F, CULPEPPER J S. The influence of topic difficulty, relevance level, and document ordering on relevance judging[C]//Proceedings of the 21st Australasian document computing symposium. New York:ACM,2016:41-48.
[33] BOOKSTEIN A. Information retrieval:a sequential learning process[J]. Journal of the American Society for information Science, 1983, 34(5):331-342.
[34] SWANSON D R. Historical note:information retrieval and the future of an illusion[J]. Journal of the American Society for Information Science, 1988, 39(2):92-98.
[35] WANG P, SOERGEL D. A cognitive model of document use during a research project. Study I. Document selection[J]. Journal of the American Society for Information Science, 1998, 49(2):115-133.
[36] WANG P, DOMAS WHITE M. A cognitive model of document use during a research project. Study II. Decisions at the reading and citing stages[J]. Journal of the American Society for Information Science, 1999, 50(2):98-114.
[37] VAKKARI P, HAKALA N. Changes in relevance criteria and problem stages in task performance[J]. Journal of documentation, 2000, 56(5):.540-562.
[38] XIE I, BENOIT E. Search result list evaluation versus document evaluation:similarities and differences[J]. Journal of documentation, 2013, 69(1):49-80.
[39] WANG Y, XIE I, LEE S. Explore eye movement patterns in search result evaluation and individual document evaluation[J]. Proceedings of the Association for Information Science and Technology, 2015, 52(1):1-4.
[40] TVERSKY A, KAHNEMAN D. Judgment under uncertainty:Heuristics and biases[J]. Science, 1974, 185(4157):1124-1131.
[41] HOGARTH R M, EINHORN H J. Order effects in belief updating:the belief-adjustment model[J]. Cognitive psychology, 1992, 24(1):1-55.
[42] TRUEBLOOD J, BUSEMEYER J. A comparison of the belief-adjustment model and the quantum inference model as explanations of order effects in human inference[C]//Proceedings of the Annual Meeting of the Cognitive Science Society. Seattle:Cognitive Science Society,2010, 32(32):1166-1171.
[43] TRUEBLOOD J S, BUSEMEYER J R. A quantum probability account of order effects in inference[J]. Cognitive science, 2011, 35(8):1518-1552.
[44] BUSEMEYER J R, WANG Z, KHRENNIKOV A, et al. Applying quantum principles to psychology[J]. Physica scripta, 2014(T163):014007.
[45] BUSEMEYER J R, KVAM P D, PLESKAC T J. Markov versus quantum dynamic models of belief change during evidence monitoring[J]. Scientific reports, 2019, 9(1):1-10.
[46] AERTS D, GABORA L, SOZZO S. Concepts and their dynamics:a quantum-theoretic modeling of human thought[J]. Topics in cognitive science, 2013, 5(4):737-772.
[47] KHRENNIKOV A. Quantum-like model of cognitive decision making and information processing[J]. Biosystems, 2009, 95(3):179-187.
[48] ZHITOMIRSKY-GEFFET M, BAR-ILAN J, LEVENE M. Categorical relevance judgment[J]. Journal of the Association for Information Science and Technology, 2018, 69(9):1084-1094.
[49] KHRENNIKOV A. The principle of supplementarity:a contextual probabilistic viewpoint to complementarity, the interference of probabilities and incompatibility of variables in quantum mechanics[J]. Foundations of physics, 2005, 35(10):1655-1693.
[50] KHRENNIKOV A. Quantum-like modeling of cognition[J]. Frontiers in physics, 2015, 3(77):1-18.
[51] TVERSKY A, KAHNEMAN D. Extensional versus intuitive reasoning:the conjunction fallacy in probability judgment[J]. Psychological review, 1983, 90(4):293.
[52] WANG Z, BUSEMEYER J R. A quantum question order model supported by empirical tests of an a priori and precise prediction[J]. Topics in cognitive science, 2013, 5(4):689-710.
[53] DZHAFAROV E N, ZHANG R, KUJALA J. Is there contextuality in behavioural and social systems?[J]. Philosophical transactions of the Royal Society A:mathematical, physical and engineering sciences, 2016, 374(2058):20150099.
[54] MOORE D W. Measuring new types of question-order effects:additive and subtractive[J]. The public opinion quarterly, 2002, 66(1):80-91.
[55] GREISDORF H. Relevance:an interdisciplinary and information science perspective[J]. Informing science, 2000, 3(2):67-72.
[56] JANES J W. Relevance judgments and the incremental presentation of document representations[J]. Information processing & management, 1991, 27(6):629-646.
[57] YEARSLEY J M, BUSEMEYER J R. Quantum cognition and decision theories:a tutorial[J]. Journal of mathematical psychology, 2016, 74:99-116.
[58] BELKIN N J. Anomalous states of knowledge as a basis of information retrieval[J]. The Canadian journal of information science, 1980, 5(1):133-143.
[59] VAKKARI P. Searching as learning:a systematization based on literature[J]. Journal of information science, 2016, 42(1):7-18.
[60] 曹树金, 李洁娜, 王志红. 面向网络信息资源聚合搜索的细粒度聚合单元元数据研究[J]. 中国图书馆学报, 2017, 43(4):74-92.
[61] 王益成,王萍,王美月.基于SVM的网络信息茧房层次敏感影响因素识别研究[J].情报资料工作, 2019,40(6):90-97.
[62] DZUNG H, LUO S. Search result diversification in resource selection for federated search[C]//International ACM SIGIR conference on research & development in information retrieval. New York:ACM, 2013:612-622.
[63] YU H, JATOWT A, BLANCO R, et al. Decoding multi-click search behavior based on marginal utility[J]. Information retrieval, 2017, 20(1):25-52.
[64] BUSEMEYER J R, WANG Z. What is quantum cognition, and how is it applied to psychology?[J]. Current directions in psychological science, 2015, 24(3):163-169.
Options
文章导航

/

〈 〉