图书情报工作 ›› 2013, Vol. 57 ›› Issue (15): 137-141.DOI: 10.7536/j.issn.0252-3116.2013.15.022

• 知识组织 • 上一篇    下一篇

一种基于类别分布信息的文本特征选择模型

刘海峰1, 于利军2, 刘守生1   

  1. 1. 解放军理工大学理学院;
    2. 解放军理工大学气象海洋学院
  • 收稿日期:2013-06-07 修回日期:2013-07-16 出版日期:2013-08-05 发布日期:2013-08-05
  • 作者简介:刘海峰,解放军理工大学理学院教授,博士,E-mail:hfliu1962@sina.com;于利军,解放军理工大学气象海洋学院硕士研究生;刘守生,解放军理工大学理学院博士。
  • 基金资助:

    本文系国家自然科学基金“直觉模糊聚类理论及其应用”(项目编号:71071161)和江苏省自然科学基金“模糊语言模型研究”(项目编号:BK2012511)研究成果之一。

An Improved TF-IDF Method of Text Feature Selection Based on Category and Frequency

Liu Haifeng1, Yu Lijun2, Liu Shousheng1   

  1. 1. Institute of sciences, PLA university of science and technology, Nanjing 210007;
    2. College of meteorology and oceanography, PLA university of science and technology, Nanjing 210022
  • Received:2013-06-07 Revised:2013-07-16 Online:2013-08-05 Published:2013-08-05

摘要:

TF-IDF是一种常用的文本特征选择方法。基于该模型的特征选择思想,以特征项的类内分布、类间分布信息为依据,通过引入类内分布及类间分布权重因子对模型的TF及IDF部分进行加权,提出一种基于类别分布信息的文本特征选择模型。新模型使得TF部分含有类内文本频数信息,同时IDF部分含有特征项的类间频数信息。随后的文本分类试验表明,平均查全率、查准率分别提高6.4%、7.8%,F1值提高约7%,验证了本研究提出的基于类别分布的文本特征选择模型的有效性。

关键词: 文本分类, 特征选择, TF-IDF, 类内分布, 类间分布

Abstract:

TF-IDF is a commonly used text feature selection method. Based on the characteristics of the model selection ideas and using the feature within class distribution and the distribution between class information as the foundations, we propose a model of text feature selection based on the category distribution information through the introduction of weighting factor distribution within classes and between classes. The new model makes the TF part contains the within class text frequency information. At the same time, the IDF part contains the between class frequency information. The subsequent text classification experiments proved that the average recall rate, precision rate increased 6.4%, 7.8% respectively. At the same time, the F1 value increased about 7%. We demonstrate the effectiveness of the text feature selection model proposed in this paper.

Key words: textcategorization, feature selection, TF-IDF, distribution within class, distribution between class

中图分类号: