图书情报工作

图书情报工作 >

2024 , Vol. 68 >Issue 23: 76 - 87

DOI: https://doi.org/10.13266/j.issn.0252-3116.2024.23.007

情报研究

基于GPT-4数据增强与对比学习的多模态谣言检测研究

  • 蒋超 ,
  • 朱学芳
展开
  • 1 南京大学信息管理学院 南京 210023;
    2 南京大学江苏省数据工程与知识服务重点实验室 南京 210023
蒋超,硕士研究生。

收稿日期: 2024-04-12

  修回日期: 2024-09-01

  网络出版日期: 2024-12-04

收起

Research on Multimodal Rumor Detection Based on GPT-4 Text Augmentation and Contrastive Learning

  • Jiang Chao ,
  • Zhu Xuefang
Expand
  • 1 School of Information Management, Nanjing University, Nanjing 210023;
    2 Jiangsu Key Laboratory of Data Engineering and Knowledge Service, Nanjing University, Nanjing 210023

Received date: 2024-04-12

  Revised date: 2024-09-01

  Online published: 2024-12-04

Fold

摘要

[目的/意义] 针对目前多模态谣言检测领域数据增强方法语义准确性与多样性受限的问题，探讨提升多模态谣言检测准确率的模型及方法，有助于进行网络谣言检测和进一步提高网络信息治理能力。[方法/过程] 提出一种提升多模态谣言检测准确率的模型TARD-GPT-4。该模型调用GPT-4进行数据增强，采用BERT与ViT模型分别获取模态特征，并使用一种有监督的对比学习策略发掘数据的标签属性特征，最后使用全连接层进行谣言检测判别。[结果/结论] 提示大语言模型使用重新描述的方式进行数据增强并融合有监督对比学习的方法对多模态谣言检测准确率的提升有着正向效果。相较于所选取的最优基线模型，TARD-GPT-4在多模态谣言检测准确率上高出1.62%；同时，通过探究多种数据增强方式对结果的影响，发现提示大语言模型使用重新描述的增强方式效果最优。

关键词： 数据增强; 对比学习; 多模态; 谣言检测

本文引用格式

蒋超 , 朱学芳 . 基于GPT-4数据增强与对比学习的多模态谣言检测研究[J]. 图书情报工作, 2024 , 68(23) : 76 -87 . DOI: 10.13266/j.issn.0252-3116.2024.23.007

Abstract

[Purpose/Significance] To enhance the semantic accuracy and diversity in data augmentation methods for multimodal rumor detection, exploring models and methods that have the potential to enhance the detection performance can contribute to the identification of online rumors, as well as to the reinforcement of network information governance capabilities.[Method/Process] A multimodal rumor detection model named TARD-GPT-4 was proposed, which leveraged GPT-4 for data augmentation.The model employed BERT and ViT models to extract textual and visual features, respectively.A supervised contrastive learning strategy was used to further explore the label attribute features.Finally, a full connected layer was used for rumor detection discrimination.[Result/Conclusion] Incorporating supervised contrastive learning and prompting large language models using rephrasing method to augment data have a positive effect on improving the accuracy of multimodal rumor detection.Compared to the optimal baseline model, TARD-GPT-4 achieves a 1.62% higher accuracy in multimodal rumor detection.The experimental part also investigates the impact of various data augmentation methods and finds that prompting LLMs for paraphrasing yields the most favorable results.

Key words： data augmentation; contrastive learning; multimodal; rumor detection

参考文献

[1] GUO B, DING Y S, YAO L N, et al. The future of false information detection on social media: new perspectives and trends[J]. ACM computing surveys, 2020, 53(4): 1-36.
[2] ALLCOTT H, GENTZKOW M. Social media and fake news in the 2016 election[J]. Journal of economic perspectives, 2017, 31(2): 211-236.
[3] KWON S, CHA M, JUNG K, et al. Prominent features of rumor propagation in online social media[C]//2013 IEEE 13th international conference on data mining. Dallas: IEEE, 2013: 1103-1108.
[4] MA J, GAO W, WONG K F, et al. Detect rumors in microblog posts using propagation structure via kernel learning[C]// Proceedings of the 55th annual meeting of the Association for Computational Linguistics. Vancouver: Association for Computational Linguistics, 2017: 708-717.
[5] CASTILLO C, MENDOZA M, POBLETE B. Information credibility on twitter[C]//Proceedings of the 20th international conference on World Wide Web. New York: Association for Computing Machinery, 2011: 675-684.
[6] YU F, LIU Q, WU S, et al. A convolutional approach for misinformation identification[C]// Proceedings of the twenty-sixth international joint conference on artificial intelligence. Melbourne: AAAI Press, 2017: 3901-3907.
[7] MA J, GAO W, MITRA P, et al. Detecting rumors from microblogs with recurrent neural networks[C]//Proceedings of the twenty-fifth international joint conference on artificial intelligence. New York: AAAI Press, 2016: 3818-3824.
[8] 凤丽洲, 刘馥榕, 王友卫. 基于图卷积网络和注意力机制的谣言检测方法[J]. 数据分析与知识发现, 2024, 8(4): 125-136. (FENG L Z, LIU F R, WANG Y W. Rumor detection method based on graph convolution network and attention mechanism[J]. Data analysis and knowledge discovery, 2024, 8(4): 125-136.)
[9] LU Y J, LI C T. GCAN: graph-aware co-attention networks for explainable fake news detection on social media[EB/OL]. [2024-9-30]. https://arxiv.org/abs/2004.11648.
[10] WANG Y, MA F, JIN Z, et al. Eann: event adversarial neural networks for multi-modal fake news detection[C]//Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. London: Association for Computing Machinery, 2018: 849-857.
[11] SINGHAL S, SHAH R R, Chakraborty T, et al. Spotfake: a multi-modal framework for fake news detection[C]//2019 IEEE fifth international conference on multimedia big data. Singapore: IEEE, 2019: 39-47.
[12] ZHANG Y, SUI E, YEUNG-LEVY S. Connect, collapse, corrupt: learning cross-modal tasks with uni-modal data[EB/OL]. [2024-9-30]. https://arxiv.org/abs/2401.08567.
[13] KHOSLA P, TETERWAK P, WANG C, et al. Supervised contrastive learning[C]//Proceedings of the 34th international conference on neural information processing systems. New York: Curran Associates, 2020: 18661-18673.
[14] KUMAR V, GLAUDE H, DE LICHY C, et al. A closer look at feature space data augmentation for few-shot intent classification[EB/OL]. [2024-09-30]. https://arxiv.org/abs/1910.04176.
[15] WEI J, ZOU K. Eda: easy data augmentation techniques for boosting performance on text classification tasks[EB/OL]. [2024-09-30]. https://arxiv.org/abs/1901.11196.
[16] FENG S Y, GANGAL V, WEI J, et al. A survey of data augmentation approaches for NLP[EB/OL]. [2024-09-30]. https://arxiv.org/abs/2105.03075.
[17] SU J L. Have it both ways: simbert model for fusion retrieval and generation[EB/OL]. [2024-09-30]. https://github.com/ZhuiyiTechnology/simbert.
[18] SANH V, DEBUT L, CHAUMOND J, et al. DistilBERT: a distilled version of bert: smaller, faster, cheaper and lighter[EB/OL]. [2024-09-30]. https://arxiv.org/abs/1910.01108.
[19] LIU Y, OTT M, GOYAL N, et al. Roberta: a robustly optimized bert pretraining approach[EB/OL]. [2024-09-30]. https://arxiv.org/abs/2406.00367.
[20] HUA J, CUI X, LI X, et al. Multimodal fake news detection through data augmentation-based contrastive learning[J]. Applied soft computing, 2023, 136(1): 110125-110133.
[21] KEYA A J, WADUD M A H, MRIDHA M F, et al. Augfake-BERT: handling imbalance through augmentation of fake news using BERT to enhance the performance of fake news classification[J]. Applied sciences, 2022, 12(17): 8398-8418.
[22] BUCOS M, ȚUCUDEAN G. Text data augmentation techniques for fake news detection in the Romanian language[J]. Applied sciences, 2023, 13(13): 7389-7408.
[23] AMJAD M, SIDOROV G, ZHILA A. Data augmentation using machine translation for fake news detection in the Urdu language[C]//Proceedings of the twelfth language resources and evaluation conference. Marseille: Language Resources Association, 2020: 2537-2542.
[24] SUYANTO S. Synonyms-based augmentation to improve fake news detection using bidirectional LSTM[C]//20208th International conference on information and communication technology. Yogyakarta: IEEE, 2020: 1-5.
[25] JIN Z, CAO J, GUO H, et al. Multimodal fusion with recurrent neural networks for rumor detection on microblogs[C]//Proceedings of the 25th ACM international conference on multimedia. New York: Association for Computing Machinery, 2017: 795-816.
[26] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. [2024-09-30]. https://arxiv.org/abs/2401.17396.
[27] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: transformers for image recognition at scale[EB/OL]. [2024-09-30]. https://arxiv.org/abs/2010.11929.
[28] HE K, FAN H, WU Y, et al. Momentum contrast for unsupervised visual representation learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. Seattle: IEEE, 2020: 9729-9738.
[29] CHEN X, HE K. Exploring simple SIAMESE representation learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. Nashville: IEEE, 2021: 15750-15758.
[30] GRILL J B, STRUB F, ALTCHÉ F, et al. Bootstrap your own latent-a new approach to self-supervised learning[C]// Proceedings of the 34th international conference on neural information processing systems. New York: Curran Associates, 2020: 21271-21284.
[31] HENDRYCKS D, GIMPEL K. Gaussian error linear units(GELUS)[EB/OL]. [2024-09-30]. https://arxiv.org/abs/2406.14854.
[32] HINTON G, SRIVASTAVA N, KRIZHEVSKY A, et al. Improving neural networks by preventing co-adaptation of feature detectors[EB/OL]. [2024-09-30]. https://arxiv.org/abs/1911.12675.
[33] AGARAP F. Deep learning using rectified linear units(RELU)[EB/OL]. [2024-09-30]. https://arxiv.org/abs/1803.08375.
[34] DAI H, LIU Z, LIAO W, et al. AugGPT: leveraging ChatGPT for text data augmentation[EB/OL]. [2024-09-30]. https://arxiv.org/abs/2302.13007.
[35] ACHIAM J, ADLER S, AGARWAL S, et al. GPT-4 technical report[EB/OL]. [2024-09-30]. https://arxiv.org/abs/2303.08774.
[36] DONG L, YANG N, WANG W, et al. Unified language model pre-training for natural language understanding and generation[C]// Proceedings of the 33rd international conference on neural information processing systems. New York: Curran Associates, 2019: 13063-13075.
[37] 王震宇, 朱学芳. 基于多模态Transformer的虚假新闻检测研究[J]. 情报学报, 2023, 42(12): 1477-1486. (WANG Z Y, ZHU X F. Research on fake news detection based on multimodal transformer[J]. Journal of the China Society for Scientific and Technical Information, 2023, 42(12): 1477-1486.)
[38] KHATTAR D, GOUD J S, GUPTA M, et al. MVAE: multimodal variational autoencoder for fake news detection[C]//The world wide web conference. New York: Association for Computing Machinery, 2019: 2915-2921.
[39] CHEN Y, LI D, ZHANG P, et al. Cross-modal ambiguity learning for multimodal fake news detection[C]//Proceedings of the ACM Web conference 2022. New York: Association for Computing Machinery, 2022: 2897-2905.
[40] CHEN T, KORNBLITH S, NOROUZI M, et al. A simple framework for contrastive learning of visual representations[C]//International conference on machine learning. Vienna: PMLR, 2020: 1597-1607.
[41] WANG F, LIU H. Understanding the behavior of contrastive loss[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. Nashville: IEEE, 2021: 2495-2504.
Options
文章导航

/

〈 〉