Library and Information Service >
Task Design in Information Retrieval Experiment：A Comparative Study on Real and Simulated Work Tasks
Received date: 2014-07-03
Revised date: 2014-07-09
Online published: 2014-08-20
Experiment is a widely used research method in information retrieval systems evaluation and information seeking behavior research. Task design is critical in the experiment because it significantly affects research results. Simulated work task situations are used frequently in experiments in recent years. However, whether it could take place of users' real work tasks affects the results of experiments to a great degree. This study conducts an experiment which recruits 42 undergraduate students. A simulated work task situation is designed and assigned to the participants to search in CNKI, who were asked to bring a real work task situation to the experiment. The paper analyzes characteristics of the two tasks in interactive information retrieval behavior and retrieval performance. Some significant differences in properties are recognized, but there are no significant effect found in users' behaviors and interactive performance when they engaged in two tasks. The paper concludes that simulated work tasks can replace real work tasks in experiment, but it needs to be carefully designed. Some related suggestions are discussed.
Li Yuelin , Xiao Xue , Hu Die . Task Design in Information Retrieval Experiment：A Comparative Study on Real and Simulated Work Tasks[J]. Library and Information Service, 2014 , 58(16) : 5 -12 . DOI: 10.13266/j.issn.0252-3116.2014.16.001
[1] Borlund P, Ingwersen P. The development of a method for the evaluation of interactive information retrieval systems[J]. Journal of Documentation, 1997,53(3):225-250.
[2] Borlund P. Experimental components for the evaluation of interactive information retrieval systems[J]. Journal of Documentation, 2000,56(1): 71-90.
[3] Borlund P, Schneider J W. Reconsideration of the simulated work task situation: A context instrument for evaluation of information retrieval interaction[EB/OL].[2013-09-16]. http://pure.iva.dk/files/31012378/Borlund%2520Schneider%2520IIiX%25202010.pdf.
[4] Blomgren L, Vallo H, Byström K. Evaluation of an information system in an information seeking process[M]//Heery R, Lyon L. Research and Advanced Technology for Digital Libraries. Berlin: Springer, 2004:57-68.
[5] Vakkari P. Task-based information searching[J]. Annual Review of Information Science and Technology, 2003,37(1):413-464.
[6] Kim J. Describing and predicting information seeking behavior on the Web[J]. Journal of the American Society for Information Science and Technology, 2009,60(4):679-693.
[7] Li Yuelin, Belkin N J. An exploration of the relationships between work task and interactive information search behavior[J]. Journal of the American Society for Information Science and Technology, 2010,61(9):1771-1789.
[8] Byström K, Järvelin K. Task complexity affects information seeking and use[J]. Information Processing & Management, 1995,31(2):191-213.
[9] Bell D J, Ruthven I. Searcher's assessments of task complexity for Web searching[C]//McDonald S, Tait J. Advances in Information Retrieval: 26th European Conference on IR Research. Berlin: Springer, 2004:57-71.
[10] Gwizdka J, Spence I. What can searching behavior tell us about the difficulty of information tasks? A study of Web navigation[J]. Proceedings of the American Society for Information Science and Technology,2006,43(1):1-22.
[11] Li Yuelin, Belkin N J. A faceted approach to conceptualizing tasks in information seeking[J]. Information Processing & Management, 2008,44(6):1822-1837.
[12] Li Yuelin. Exploring the relationships between work task and search task in information search[J]. Journal of the American Society for Information Science and Technology, 2009,60(2):275-291.
[13] Liu Jingjing,Belkin N J. Personalizing information retrieval for multi-session tasks: The roles of task stage and task type[C]//Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval.New York:ACM, 2010: 26-33.
[14] Liu Jingjing,Gwizdka J, Liu Chang, et al. Predicting task difficulty for different task types[J]. Proceedings of the American Society for Information Science and Technology, 2010,47(1):1-10.
[15] Xie Iris. Dimensions of tasks: Influences on information-seeking and retrieving processes[J]. Journal of Documentation,2009,65(3):339-366.
[16] Li Yuelin. Investigating the relationships between facets of work task and selection and query-related behavior[J]. Chinese Journal of Library and Information Science, 2012,5(1):51-69.
[17] Yuan Xiaojun, Belkin N J. Investigating information retrieval support techniques for different information-seeking strategies[J]. Journal of the American Society for Information Science and Technology, 2010,61(8):1543-1563.
[18] Yuan Xiaojun, Belkin N J. Evaluating an integrated system supporting multiple information-seeking strategies[J]. Journal of the American Society for Information Science and Technology, 2010,61(10):1987-2010.
[19] Maynard D C,Hakel M D. Effects of objective and subjective task complexity on performance[J]. Human Performance,1997,10(4):303-330.
[20] Liu Jingjing, Liu Chang, Yuan Xiaojun, et al. Understanding searchers' perception of task difficulty:Relationships with task type[J]. Proceedings of the American Society for Information Science and Technology,2011,48(1):1-10.
[21] Li Yuelin. An exploration of the relationships between work tasks and users' interaction performance[J]. Proceedings of the American Society for Information Science and Technology,2010,47(1):1-9.
/
| 〈 |
|
〉 |
