文本增强与预训练语言模型在网络问政留言分类中的集成对比研究

doi:10.13266/j.issn.0252-3116.2021.13.010

图书情报工作 ›› 2021, Vol. 65 ›› Issue (13): 96-107.DOI: 10.13266/j.issn.0252-3116.2021.13.010

文本增强与预训练语言模型在网络问政留言分类中的集成对比研究

施国良, 陈宇奇

河海大学商学院南京 211100

收稿日期:2020-10-07 修回日期:2021-03-19 出版日期:2021-07-05 发布日期:2021-07-10
作者简介:施国良(ORCID:0000-0001-7585-640X),副教授,博士,E-mail:shigl@hhu.edu.cn;陈宇奇(ORCID:0000-0001-5755-5208),硕士研究生。
基金资助:
本文系中央高校基本业务费项目"基于图数据库的水利知识图谱关键技术研究"（项目编号：B200207036）研究成果之一。

A Comparative Study on the Integration of Text Enhanced and Pre-trained Language Models in the Classification of Internet Political Messages

Shi Guoliang, Chen Yuqi

Business School, Hohai University, Nanjing 211100

Received:2020-10-07 Revised:2021-03-19 Online:2021-07-05 Published:2021-07-10

摘要/Abstract

摘要： [目的/意义] 政府网络问政平台是政府部门知晓民意的重要途径之一，为提高问政留言分类的精度以及处理留言数据质量差、数量少等问题，对比多种基于BERT改进模型与文本增强技术结合的分类效果并探究其差异原因。[方法/过程] 设计网络问政留言分类集成对比模型，文本增强方面采用EDA技术与SimBERT文本增强技术进行对比实验，文本分类模型方面则采用多种基于BERT改进的预训练语言模型（如ALBERT、RoBERTa）进行对比实验。[结果/结论] 实验结果表明，基于RoBERTa与SimBERT文本增强的文本分类模型效果最佳，在测试集上的F1值高达92.05%，相比于未进行文本增强的BERT-base模型高出2.89%。同时，SimBERT文本增强后F1值相比未增强前平均提高0.61%。实验证明了基于RoBERTa与SimBERT文本增强模型能够有效提升多类别文本分类的效果，在解决同类问题时具有较强可借鉴性。

关键词: 问政平台, 文本分类, 文本增强, BERT模型

Abstract: [Purpose/significance] Government network platform for political inquiry is one of the important ways for rulers to know public opinions. In order to improve the accuracy of the classification of political inquiry messages and to deal with the problems such as poor quality and small quantity of message data, the classification effects of various BERT improved models combined with text enhancement technology and the reasons for their differences were explored.[Method/process] Design the network political inquiry message classification integrated comparison model,the EDA (Easier Data Augment) technology and SimBERT text Augment technology were used for comparison experiment in the aspect of text augmentation, and various pre-training language models (such as ALBERT and RoBERTa) based on BERT improvement were used for comparison experiment in the aspect of text classification model.[Result/conclusion] The experimental results showed that the text classification model based on RoBERTa and SimBERT text enhancement had the best effect, and the F1 value on the test set was as high as 92.05%, 2.89% higher than that of the Bert-Base model without text enhancement. At the same time, F1 value after SimBERT text enhancement was 0.61% higher than that before no enhancement. The experiment proved that text enhancement model based on RoBERTa and SimBERT can effectively improve the classification effect of multiple categories of text classification problems, and has strong referability in solving similar problems.

Key words: political platform, text classification, text enhancement, BERT model

中图分类号:

G254

施国良, 陈宇奇. 文本增强与预训练语言模型在网络问政留言分类中的集成对比研究[J]. 图书情报工作, 2021, 65(13): 96-107.

Shi Guoliang, Chen Yuqi. A Comparative Study on the Integration of Text Enhanced and Pre-trained Language Models in the Classification of Internet Political Messages[J]. Library and Information Service, 2021, 65(13): 96-107.

参考文献

[1] 徐晓雯,曹守新.网络问政对公共政策制定的影响——基于SWOT分析方法[J].山东社会科学,2015(6):179-183.
[2] 马思丹,刘东苏.基于加权Word2vec的文本分类方法研究[J].情报科学,2019,37(11):38-42.
[3] 程婧,刘娜娜,闵可锐,等.一种低频词词向量优化方法及在短文本分类中的应用[J].计算机科学,2020(8):255-260.
[4] PETERS M E, NEUMANN M, IYYER M, et al. Deep contextualized word representations[C]//Proceedings of the conference of the north american chapter of the Association for Computational Linguistics:human language technologies. Stroudsburg:Association for Computational Linguistics, 2018:2227-2237.
[5] DEVLIN J, CHANG M W, LEE K, et al. BERT:pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the conference of the North American chapter of the Association for Computational Linguistics:human language technologies. Stroudsburg:Association for Computational Linguistics, 2019:4171-4186.
[6] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceeding of advances in neural information processing systems. California:MIT Press, 2017:6000-6010.
[7] 陈燕方.基于DDAG-SVM的在线商品评论可信度分类模型[J].情报理论与实践,2017,40(7):132-137.
[8] 余本功,曹雨蒙,陈杨楠,等.基于nLD-SVM-RF的短文本分类研究[J].数据分析与知识发现,2020,4(1):111-120.
[9] 韩栋,王春华,肖敏.基于句子级学习改进CNN的短文本分类方法[J].计算机工程与设计,2019,40(1):256-260.
[10] 杨云龙,孙建强,宋国超.基于GRU和胶囊特征融合的文本情感分析[J/OL].[2021-02-10]. http://kns.cnki.net/kcms/detail/51.1307.TP.20200429.1704.010.html.
[11] 赵旸,张智雄,刘欢,等.基于BERT模型的中文医学文献分类研究[J].数据分析与知识发现, 2020(8):41-49.
[12] 吴俊,程垚,郝瀚,等.基于BERT嵌入BiLSTM-CRF模型的中文专业术语抽取研究[J].情报学报,2020,39(4):409-418.
[13] 廖胜兰,吉建民,俞畅,等.基于BERT模型与知识蒸馏的意图分类方法[J/OL].[2021-02-10]. https://doi.org/10.19678/j.issn.1000-3428.0057416.
[14] JASON W, KAI Z. Eda:easy data augmentation techniques for boosting performance on text classification tasks[C]//Proceeding of the 2019 conference on empirical methods in natural language processing. Hong Kong:ACL, 2019.
[15] 俞畅,欧阳昱,张波,等.基于对抗式生成网络的电力用户意图文本生成[J].信息技术与网络安全,2019,38(11):67-72.
[16] LIU Y, OTT M, GOYAL N, et al. RoBERTa:a robustly optimized BERT pretraining approach[J/OL].[2021-02-10].https://arxiv.org/abs/1907.11692.
[17] LAN Z Z, CHEN M D, GOODMAN S, et al. ALBERT:a lite BERT for self-supervised learning of language representations[C]//Proceedings of the international conference on learning representations. Ethiopia:ICLR, 2020.
[18] 苏剑林.鱼与熊掌兼得:融合检索和生成的SimBERT模型[EB/OL].[2020-05-18]. https://kexue.fm/archives/7427.
[19] LIU P F, QIU X P, HUANG X J. Recurrent neural for text classification with multi-task learning[C]//Proceeding of the international joint conference on artificial intelligence. New York:IJCAI, 2016.
[20] KIM Y. Convolutional neural networks for sentence classification[C]//Proceeding of the 2014 conference on empirical methods in natural language processing. Doha:ACM, 2014.
[21] LAI S W, XU L H, LIU K, et al. Recurrent convolutional neural networks for text classification[C]//Proceeding of the 29th national conference on artificial intelligence. Austin:AAAI, 2015.
[22] YANG Z L, DAI Z H, YANG Y Y, et al. XLNet:generalized autoregressive pretraining for language understanding[C]//Proceeding of the 33rd conference on neural information processing systems. Vancouver:MIT Press, 2019.
[23] SANH V, DEBUT L, CHAUMOND J, et al. DistilBERT, a distilled version of BERT:smaller,faster,cheaper and lighter[J/OL].[2021-02-10].https://arxiv.org/abs/1910.01108v4.
[24] JIAO X Q, YIN Y C, SHANG L F, et al. TinyBERT:distilling BERT for natural language understanding[J/OL].[2021-02-10].https://arxiv.org/abs/1909.10351v3.
[25] BAO H B, DONG L, WEI F R, et al. UniLMv2:pseudo-masked language models for unified language model pre-trainning[J/OL].[2021-02-10].https://arxiv.org/abs/2002.12804.

文本增强与预训练语言模型在网络问政留言分类中的集成对比研究

A Comparative Study on the Integration of Text Enhanced and Pre-trained Language Models in the Classification of Internet Political Messages

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	宗威, 林松涛, 刘继昶. 数据集成场景下考虑语义信息的高效实体识别方法[J]. 图书情报工作, 2022, 66(14): 128-136.
[2]	张玉洁, 白如江, 刘明月, 于纯良. 融合语义联想和BERT的图情领域SAO短文本分类研究[J]. 图书情报工作, 2021, 65(16): 118-129.
[3]	刘忠宝, 党建飞, 张志剑. 《史记》历史事件自动抽取与事理图谱构建研究[J]. 图书情报工作, 2020, 64(11): 116-124.
[4]	王文娟, 马建霞, 陈春, 张凌波. 引文文本分类与实现方法研究综述[J]. 图书情报工作, 2016, 60(6): 118-127.
[5]	李晓军, 刘怀亮, 杜坤. 一种基于复杂网络模型的作者身份识别方法[J]. 图书情报工作, 2015, 59(18): 102-107.
[6]	尹丽英, 赵捧未. 基于语义网络社团划分的中文文本分类研究[J]. 图书情报工作, 2014, 58(19): 124-128.
[7]	陆泉, 陈静, 丁恒. 基于社会标签的图像情感自动分类标注研究[J]. 图书情报工作, 2014, 58(12): 118-123.
[8]	马甲林, 刘金岭, 金春霞. 基于概念簇的文本分类算法[J]. 图书情报工作, 2013, 57(15): 132-136,82.
[9]	刘海峰, 于利军, 刘守生. 一种基于类别分布信息的文本特征选择模型[J]. 图书情报工作, 2013, 57(15): 137-141.
[10]	薛春香, 张玉芳. 面向新闻领域的中文文本分类研究综述[J]. 图书情报工作, 2013, 57(14): 134-139.
[11]	赵辉, 刘怀亮. 一种基于维基百科的中文短文本分类算法[J]. 图书情报工作, 2013, 57(11): 120-124.
[12]	路永和, 李焰锋. 改进TF-IDF算法的文本特征项权值计算方法[J]. 图书情报工作, 2013, 57(03): 90-95.
[13]	秦宝宝, 宋继伟, 董尹, 牛青, 吕美香, 陈彬, 李骁. 竞争情报系统中一种自动文本分类策略——以民用航空客服行业为例[J]. 图书情报工作, 2012, 56(24): 39-43,119.
[14]	姚长青, 杜永萍. 基于主题的舆情跟踪方法研究及性能评价[J]. 图书情报工作, 2012, 56(18): 50-53,109.
[15]	周丽红刘勘. 基于关联规则的科技文献分类研究[J]. 图书情报工作, 2012, 56(04): 12-119.