INFORMATION RESEARCH

Research on Multimodal Rumor Detection Based on GPT-4 Text Augmentation and Contrastive Learning

  • Jiang Chao ,
  • Zhu Xuefang
Expand
  • 1 School of Information Management, Nanjing University, Nanjing 210023;
    2 Jiangsu Key Laboratory of Data Engineering and Knowledge Service, Nanjing University, Nanjing 210023

Received date: 2024-04-12

  Revised date: 2024-09-01

  Online published: 2024-12-04

Abstract

[Purpose/Significance] To enhance the semantic accuracy and diversity in data augmentation methods for multimodal rumor detection, exploring models and methods that have the potential to enhance the detection performance can contribute to the identification of online rumors, as well as to the reinforcement of network information governance capabilities.[Method/Process] A multimodal rumor detection model named TARD-GPT-4 was proposed, which leveraged GPT-4 for data augmentation.The model employed BERT and ViT models to extract textual and visual features, respectively.A supervised contrastive learning strategy was used to further explore the label attribute features.Finally, a full connected layer was used for rumor detection discrimination.[Result/Conclusion] Incorporating supervised contrastive learning and prompting large language models using rephrasing method to augment data have a positive effect on improving the accuracy of multimodal rumor detection.Compared to the optimal baseline model, TARD-GPT-4 achieves a 1.62% higher accuracy in multimodal rumor detection.The experimental part also investigates the impact of various data augmentation methods and finds that prompting LLMs for paraphrasing yields the most favorable results.

Cite this article

Jiang Chao , Zhu Xuefang . Research on Multimodal Rumor Detection Based on GPT-4 Text Augmentation and Contrastive Learning[J]. Library and Information Service, 2024 , 68(23) : 76 -87 . DOI: 10.13266/j.issn.0252-3116.2024.23.007

References

[1] GUO B, DING Y S, YAO L N, et al. The future of false information detection on social media: new perspectives and trends[J]. ACM computing surveys, 2020, 53(4): 1-36.
[2] ALLCOTT H, GENTZKOW M. Social media and fake news in the 2016 election[J]. Journal of economic perspectives, 2017, 31(2): 211-236.
[3] KWON S, CHA M, JUNG K, et al. Prominent features of rumor propagation in online social media[C]//2013 IEEE 13th international conference on data mining. Dallas: IEEE, 2013: 1103-1108.
[4] MA J, GAO W, WONG K F, et al. Detect rumors in microblog posts using propagation structure via kernel learning[C]// Proceedings of the 55th annual meeting of the Association for Computational Linguistics. Vancouver: Association for Computational Linguistics, 2017: 708-717.
[5] CASTILLO C, MENDOZA M, POBLETE B. Information credibility on twitter[C]//Proceedings of the 20th international conference on World Wide Web. New York: Association for Computing Machinery, 2011: 675-684.
[6] YU F, LIU Q, WU S, et al. A convolutional approach for misinformation identification[C]// Proceedings of the twenty-sixth international joint conference on artificial intelligence. Melbourne: AAAI Press, 2017: 3901-3907.
[7] MA J, GAO W, MITRA P, et al. Detecting rumors from microblogs with recurrent neural networks[C]//Proceedings of the twenty-fifth international joint conference on artificial intelligence. New York: AAAI Press, 2016: 3818-3824.
[8] 凤丽洲, 刘馥榕, 王友卫. 基于图卷积网络和注意力机制的谣言检测方法[J]. 数据分析与知识发现, 2024, 8(4): 125-136. (FENG L Z, LIU F R, WANG Y W. Rumor detection method based on graph convolution network and attention mechanism[J]. Data analysis and knowledge discovery, 2024, 8(4): 125-136.)
[9] LU Y J, LI C T. GCAN: graph-aware co-attention networks for explainable fake news detection on social media[EB/OL]. [2024-9-30]. https://arxiv.org/abs/2004.11648.
[10] WANG Y, MA F, JIN Z, et al. Eann: event adversarial neural networks for multi-modal fake news detection[C]//Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. London: Association for Computing Machinery, 2018: 849-857.
[11] SINGHAL S, SHAH R R, Chakraborty T, et al. Spotfake: a multi-modal framework for fake news detection[C]//2019 IEEE fifth international conference on multimedia big data. Singapore: IEEE, 2019: 39-47.
[12] ZHANG Y, SUI E, YEUNG-LEVY S. Connect, collapse, corrupt: learning cross-modal tasks with uni-modal data[EB/OL]. [2024-9-30]. https://arxiv.org/abs/2401.08567.
[13] KHOSLA P, TETERWAK P, WANG C, et al. Supervised contrastive learning[C]//Proceedings of the 34th international conference on neural information processing systems. New York: Curran Associates, 2020: 18661-18673.
[14] KUMAR V, GLAUDE H, DE LICHY C, et al. A closer look at feature space data augmentation for few-shot intent classification[EB/OL]. [2024-09-30]. https://arxiv.org/abs/1910.04176.
[15] WEI J, ZOU K. Eda: easy data augmentation techniques for boosting performance on text classification tasks[EB/OL]. [2024-09-30]. https://arxiv.org/abs/1901.11196.
[16] FENG S Y, GANGAL V, WEI J, et al. A survey of data augmentation approaches for NLP[EB/OL]. [2024-09-30]. https://arxiv.org/abs/2105.03075.
[17] SU J L. Have it both ways: simbert model for fusion retrieval and generation[EB/OL]. [2024-09-30]. https://github.com/ZhuiyiTechnology/simbert.
[18] SANH V, DEBUT L, CHAUMOND J, et al. DistilBERT: a distilled version of bert: smaller, faster, cheaper and lighter[EB/OL]. [2024-09-30]. https://arxiv.org/abs/1910.01108.
[19] LIU Y, OTT M, GOYAL N, et al. Roberta: a robustly optimized bert pretraining approach[EB/OL]. [2024-09-30]. https://arxiv.org/abs/2406.00367.
[20] HUA J, CUI X, LI X, et al. Multimodal fake news detection through data augmentation-based contrastive learning[J]. Applied soft computing, 2023, 136(1): 110125-110133.
[21] KEYA A J, WADUD M A H, MRIDHA M F, et al. Augfake-BERT: handling imbalance through augmentation of fake news using BERT to enhance the performance of fake news classification[J]. Applied sciences, 2022, 12(17): 8398-8418.
[22] BUCOS M, ȚUCUDEAN G. Text data augmentation techniques for fake news detection in the Romanian language[J]. Applied sciences, 2023, 13(13): 7389-7408.
[23] AMJAD M, SIDOROV G, ZHILA A. Data augmentation using machine translation for fake news detection in the Urdu language[C]//Proceedings of the twelfth language resources and evaluation conference. Marseille: Language Resources Association, 2020: 2537-2542.
[24] SUYANTO S. Synonyms-based augmentation to improve fake news detection using bidirectional LSTM[C]//20208th International conference on information and communication technology. Yogyakarta: IEEE, 2020: 1-5.
[25] JIN Z, CAO J, GUO H, et al. Multimodal fusion with recurrent neural networks for rumor detection on microblogs[C]//Proceedings of the 25th ACM international conference on multimedia. New York: Association for Computing Machinery, 2017: 795-816.
[26] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. [2024-09-30]. https://arxiv.org/abs/2401.17396.
[27] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: transformers for image recognition at scale[EB/OL]. [2024-09-30]. https://arxiv.org/abs/2010.11929.
[28] HE K, FAN H, WU Y, et al. Momentum contrast for unsupervised visual representation learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. Seattle: IEEE, 2020: 9729-9738.
[29] CHEN X, HE K. Exploring simple SIAMESE representation learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. Nashville: IEEE, 2021: 15750-15758.
[30] GRILL J B, STRUB F, ALTCHÉ F, et al. Bootstrap your own latent-a new approach to self-supervised learning[C]// Proceedings of the 34th international conference on neural information processing systems. New York: Curran Associates, 2020: 21271-21284.
[31] HENDRYCKS D, GIMPEL K. Gaussian error linear units(GELUS)[EB/OL]. [2024-09-30]. https://arxiv.org/abs/2406.14854.
[32] HINTON G, SRIVASTAVA N, KRIZHEVSKY A, et al. Improving neural networks by preventing co-adaptation of feature detectors[EB/OL]. [2024-09-30]. https://arxiv.org/abs/1911.12675.
[33] AGARAP F. Deep learning using rectified linear units(RELU)[EB/OL]. [2024-09-30]. https://arxiv.org/abs/1803.08375.
[34] DAI H, LIU Z, LIAO W, et al. AugGPT: leveraging ChatGPT for text data augmentation[EB/OL]. [2024-09-30]. https://arxiv.org/abs/2302.13007.
[35] ACHIAM J, ADLER S, AGARWAL S, et al. GPT-4 technical report[EB/OL]. [2024-09-30]. https://arxiv.org/abs/2303.08774.
[36] DONG L, YANG N, WANG W, et al. Unified language model pre-training for natural language understanding and generation[C]// Proceedings of the 33rd international conference on neural information processing systems. New York: Curran Associates, 2019: 13063-13075.
[37] 王震宇, 朱学芳. 基于多模态Transformer的虚假新闻检测研究[J]. 情报学报, 2023, 42(12): 1477-1486. (WANG Z Y, ZHU X F. Research on fake news detection based on multimodal transformer[J]. Journal of the China Society for Scientific and Technical Information, 2023, 42(12): 1477-1486.)
[38] KHATTAR D, GOUD J S, GUPTA M, et al. MVAE: multimodal variational autoencoder for fake news detection[C]//The world wide web conference. New York: Association for Computing Machinery, 2019: 2915-2921.
[39] CHEN Y, LI D, ZHANG P, et al. Cross-modal ambiguity learning for multimodal fake news detection[C]//Proceedings of the ACM Web conference 2022. New York: Association for Computing Machinery, 2022: 2897-2905.
[40] CHEN T, KORNBLITH S, NOROUZI M, et al. A simple framework for contrastive learning of visual representations[C]//International conference on machine learning. Vienna: PMLR, 2020: 1597-1607.
[41] WANG F, LIU H. Understanding the behavior of contrastive loss[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. Nashville: IEEE, 2021: 2495-2504.
Outlines

/