针对主题爬行技术中的单一分类算法在面对多主题Web抓取和分类需求时泛化能力不强的局限,设计一种利用多种强分类算法形成的分类器组合,主题爬行器根据当前主题任务在线评估并为分类器排名,从中选择最优分类器分类的策略,并开展在多个主题抓取任务下的分类实验,比较每种分类算法的准确率和组合后的平均分类准确率以及对分类效率等评价指标的综合分析,结果证明该策略对领域局域性有所克服,普适性较强。
For the limitation that generalization capacity of single classification algorithm is not strong when focused crawler is facing multi-topic Web crawling and classification, the paper proposed a strategy of using multi-classifier combination formed of multiple strong classification algorithms. The focused crawler evaluates and ranks the classifiers online according to the current topic, and classifies Web pages by selecting the better classifiers. Through classification experiments of multiple topics crawling tasks, comparing between accurate rate of each classification algorithm and average classification accurate rate of multi-classifier combination, and comprehensive analysis of the two indicators——classification accuracy and classification efficiency, it proved the proposed method is better in universality, to a certain extent and overcomes the limitations of a single classifier.
[1] Mitchell T M.Machine learning[M].Columbus:The McGraw-Hill Companies Inc, 1997.
[2] Nigam K, McCallum A, Thrun S, et al. Text classification from labeled and unlabelled documents using EM[J]. Machine Learning, 2000, 39(2-3):103-134.
[3] Rennie J, MeCallum A. Using reinforcement learning to spider the Web efficiently [C]//Bratko I, Dzeroski S. Proceedings of the Sixteenth International Conference on Machine Learning. San Francisco:Morgan Kaufmann Publishers Inc, 1999:335-343.
[4] Diligenti M, Coetzee F, Lawrence S.Focused crawling using context graphs[C]//Abbadi A E, Brodie M L, Chakravarthy S, et al. Proceedings of the 26th VLDB Conference.San Francisco:Morgan Kaufmann Publishers Inc, 2000:527-534.
[5] Chakrabarti S, Punera K, Subramanyam M. Accelerated focused crawling through online relevance feedback[C]//Proceedings of the 11th International Conference on World Wide Web. New York:ACM,2002:148-159.
[6] Johnson J, Tsioutsiouliklis K, Giles C L. Evolving strategies for focused Web crawling[C]//Fawcett T,Mishra N.Proceedings of the Twentieth International Conference. Washington D.C:AAAI Press, 2003:298-305.
[7] Pant G, Srinivasan P. Link contexts in classifier-guided topical crawlers[J]. IEEE Transactions on Knowledge and Data Engineering, 2006, 18 (1):107-122.
[8] Pant G, Srinivasan P. Learning to crawl:Comparing classification schemes[J]. ACM Trans Information Systems, 2005, 23 (4):430-462.
[9] 杨建良,王永成.自动分类技术的发展与展望[EB/OL]. [2013-06-06].http://www.cnindex.fudan.edu.cn/zgsy/2003n1/zidongfenlei.htm.
[10] 张超群.基于网页分块技术的主题爬行[D].长春:吉林大学,2007.
[11] 刘菊新,徐从富.基于多分类器组合模型的垃圾邮件过滤[J].计算机工程, 2010(18):194- 196.
[12] 侯帅,韩中庚,黄洁,等.基于Sugeno模糊积分的多分类器融合方法在多属性决策中的应用[J].信息工程大学学报,2010(1):124-128.
[13] 陈冰,张化祥.集成学习的多分类器动态组合方法[J].计算机工程,2008(24):218-220.
[14] Machine learning group at university of waikato.Weka 3:Data mining software in java[EB/OL]. [2013-06-06].http://www.cs.waikato.ac.nz/ml/weka/.
[15 ] Pazzani M J, Billsus D. Learning and revising user profiles:The identification of interesting Web sites[J].Machine Learning,1997, 27(3):313-331.
[16 ] Sergej S, Martin T, Stefan S. BINGO!:Bookmark-induced gathering of information[C]//Wang Ling Tok, Dayal U, Bertino E. Proceedings of the 3rd International Conference on Web Information Systems Engineering.Los Alamitos:IEEE Computer Society, 2002:323-332.
[17 ] Miller R. WebSPHINX:A personal, customizable Web crawler [EB/OL]. [2011-02-12]. http://www.cs.cmu.edu/~rcm/websphinx/.
