INFORMATION RESEARCH

Research on Similar Patent Identification Based on Multimodal Feature Fusion

  • Xie Xiaodong ,
  • Wu Jie ,
  • Sheng Yongxiang ,
  • Wang Jiangang ,
  • Zhou Xiao
Expand
  • School of Economics and Management, Jiangsu University of Science and Technology, Zhenjiang 212003

Received date: 2024-01-15

  Revised date: 2024-04-12

  Online published: 2024-10-08

Supported by

This work is supported by the general program of National Natural Science Foundation of China titled “Resilience of Industrial Innovation Ecosystems for Industrial Safety: Implications, Assessment, and Optimization Strategies” (Grant No. 72171122), and Postgraduate Research and Practice Innovation Program of Jiangsu Province titled “Research on the Selection of Potential Partners and Collaboration Directions for Innovation Consortia” (Grant No. KYCX23_3817).

Abstract

[Purpose/Significance] The burgeoning number of patents poses significant challenges to patent retrieval, highlighting the urgent need for advanced computational techniques to identify similar patents. [Method/Process] This paper proposed a multimodal feature fusion method for similar patent identification. It utilized the BERT-wwm model and the ResNet-50 model to extract textual and image features of patents, respectively. By integrating self-attention and cross-attention mechanisms, the method effectively harnessed intra-modal feature information and inter-modal interaction information. Based on these, the model was trained and optimized for the similar patent identification. [Result/Conclusion] Empirical tests using IPC category “C08F10/00” data demonstrate that the model achieves an accuracy of 80.03% and a recall rate of 82.01%, outperforming baseline models. In simulations of similar patent identification, the model reaches a recall rate of 88.89%, indicating superior practical performance. The fusion of textual and image modal features significantly enhances the accuracy and efficiency of similar patent identification. This approach facilitates improved patent retrieval efficiency, accelerates the patent examination process, aids in patent alert analysis, and strengthens intellectual property protection.

Cite this article

Xie Xiaodong , Wu Jie , Sheng Yongxiang , Wang Jiangang , Zhou Xiao . Research on Similar Patent Identification Based on Multimodal Feature Fusion[J]. Library and Information Service, 2024 , 68(18) : 112 -122 . DOI: 10.13266/j.issn.0252-3116.2024.18.011

References

[1] 王曰芬, 谢寿峰, 邱玉婷. 面向预警的专利文献相似度研究的意义及现状 [J]. 情报理论与实践, 2014, 37(7): 135-140. (WANG Y F, XIE S F, QIU Y T. Significance and current status of the research on patent similarity computation for early warning [J]. Information studies: theory & application, 2014, 37(7): 135-140.)
[2] 赖院根, 朱东华. 专利预警警情的理论研究 [J]. 科学学与科学技术管理, 2009, 30(2): 5-9. (LAI Y G, ZHU D H. Theoretical research on alert-situation of patent early-warning system [J]. Science of science and management of S.&.T., 2009, 30(2): 5-9.)
[3] KIM S, YOON B. Patent infringement analysis using a text mining technique based on SAO structure [J]. Computers in industry, 2021, 125: 103379.
[4] PARK I, YOON B. A semantic analysis approach for identifying patent infringement based on a product–patent map [J]. Technology analysis & strategic management, 2014, 26(8): 855-874.
[5] ZHU D. Bibliometric analysis of patent infringement retrieval model based on self-organizing map neural network algorithm [J]. Library hi tech, 2020, 38(2): 479-491.
[6] LEE C, SONG B, PARK Y. How to assess patent infringement risks: a semantic patent claim analysis using dependency relationships [J]. Technology analysis & strategic management, 2013, 25(1): 23-38.
[7] CHEN L, XU S, ZHU L, et al. A deep learning based method for extracting semantic information from patent documents [J]. Scientometrics, 2020, 125(1): 289-312.
[8] LIN W, YU W, XIAO R. Measuring patent similarity based on text mining and image recognition [J]. Systems, 2023, 11(6): 294.
[9] 张杰, 孙宁宁, 张海超, 等. 基于SAO结构的中文相似专利识别算法及其应用 [J]. 情报学报, 2016, 35(5): 472-482. (ZHANG J, SUN N N, ZHANG H C, et al. Method and application of Chinese similar patents recognition based on SAO structures [J]. Journal of the China Society for Scientific and Technical Information, 2016, 35(5): 472-482.)
[10] WANG X, REN H, CHEN Y, et al. Measuring patent similarity with SAO semantic analysis [J]. Scientometrics, 2019, 121(1): 1-23.
[11] LEE S, YOON B, PARK Y. An approach to discovering new technology opportunities: Keyword-based patent map approach [J]. Technovation, 2009, 29(6-7): 481-497.
[12] LEE C, CHO Y, SEOL H, et al. A stochastic patent citation analysis approach to assessing future technological impacts [J]. Technological forecasting and social change, 2012, 79(1): 16-29.
[13] 吕学强, 罗艺雄, 李家全, 等. 中文专利侵权检测研究综述 [J]. 数据分析与知识发现, 2021, 5(3): 60-68. (LV X Q, LUO Y X, LI J Q, et al. Review of studies on detecting Chinese patent infringements [J]. Data analysis and knowledge discovery, 2021, 5(3): 60-68.)
[14] 马文姗, 赵海宁, 翟东升. 中文专利侵权检索模型研究 [J]. 情报杂志, 2012, 31(4): 175-179,195. (MA W S, ZHAO H N, ZHAI D S. Research on Chinese patent infringement retrieval model [J]. Journal of intelligence, 2012, 31(4): 175-179,195.)
[15] 彭继东, 谭宗颖. 一种基于文本挖掘的专利相似度测量方法及其应用 [J]. 情报理论与实践, 2010, 33(12): 114-118. (PENG J D, TAN Z Y. A text mining-based patent similarity measurement method and its application [J]. Information studies: theory & application, 2010, 33(12): 114-118.)
[16] 俞琰, 陈磊, 姜金德, 等. 结合词向量和统计特征的专利相似度测量方法 [J]. 数据分析与知识发现, 2019, 3(9): 53-59. (YU Y, CHEN L, JIANG J D, et al. Measuring patent similarity with word embedding and statistical features [J]. Data analysis and knowledge discovery, 2019, 3(9): 53-59.)
[17] YOON B U, YOON C B, PARK Y T. On the development and application of a self–organizing feature map–based patent map [J]. R&D Management, 2002, 32(4): 291-300.
[18] 曹祺, 赵伟, 张英杰, 等. 基于Doc2Vec的专利文件相似度检测方法的对比研究 [J]. 图书情报工作, 2018, 62(13): 74-81. (CAO Q, ZHAO W, ZHANG Y J, et al. Comparative study of patent documents similarity detection on deep learning of Doc2Vec based methods [J]. Library and information service, 2018, 62(13): 74-81.)
[19] BALTRUŠAITIS T, AHUJA C, MORENCY L P. Multimodal machine learning: a survey and taxonomy [J]. IEEE transactions on pattern analysis and machine intelligence, 2019, 41(2): 423-443.
[20] ATREY P K, HOSSAIN M A, EL SADDIK A, et al. Multimodal fusion for multimedia analysis: a survey [J]. Multimedia systems, 2010, 16(6): 345-379.
[21] JABEEN S, LI X, AMIN M S, et al. A review on methods and applications in multimodal deep learning [J]. ACM trans. multimed comput. commun. appl, 2023, 19(2s): 1-41.
[22] CAI G, XIA B. Convolutional neural networks for multimedia sentiment analysis; proceedings of the natural language processing and Chinese computing[C]//Natural language processing and chinese computing, Springer, Cham, 2015: 159-167.
[23] HUANG F, ZHANG X, ZHAO Z, et al. Image–text sentiment analysis via deep multimodal attentive fusion [J]. Knowledge-based systems, 2019, 167: 26-37.
[24] CHEN J, WANG C, WANG K, et al. HEU Emotion: a large-scale database for multimodal emotion recognition in the wild [J]. Neural computing and applications, 2021, 33(14): 8669-8685.
[25] HUANG Y, YANG X, GAO J, et al. Knowledge-driven egocentric multimodal activity recognition [J]. ACM trans. multimed comput. commun. appl, 2020, 16(4): 1-133.
[26] DANG-NGUYEN D-T, PIRAS L, GIACINTO G, et al. Multimodal retrieval with diversification and relevance feedback for tourist attraction images [J]. ACM trans. multimed comput. commun. appl, 2017, 13(4): 1-24.
[27] SENGUPTA S, BASAK S, SAIKIA P, et al. A review of deep learning with special emphasis on architectures, applications and recent trends [J]. Knowledge-based systems, 2020, 194: 105596.
[28] JIANG S, HU J, MAGEE C L, et al. Deep learning for technical document classification [J]. IEEE transactions on engineering management, 2024, 71: 1163-1179.
[29] FANG L, ZHANG L, WU H, et al. Patent2Vec: Multi-view representation learning on patent-graphs for patent classification [J]. World wide web, 2021, 24(5): 1791-1812.
[30] 李晴晴, 周长胜, 吕学强, 等. 基于外观设计专利的多模态图像检索 [J]. 计算机工程与设计, 2016, 37(9): 2469-2474. (LI Q Q, ZHOU C S, LV X Q, et al. Multi-moda limage retrieval based on design patent image [J]. Computer engineering and design, 2016, 37(9): 2469-2474.)
[31] PUSTU-IREN K, BRUNS G, EWERTH R. A multimodal approach for semantic patent image retrieval [C]// Proc. 2nd workshop patent text mining semantic technol. co-located 44th Int. ACM SIGIR Conf. Res. Develop. Inf. Retr. (SIGIR), 2021: 45-49.
[32] WEI X, ZHANG T, LI Y, et al. Multi-modality cross attention network for image and sentence matching [C]// Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. Seattle: IEEE, 2020: 10941-10950.
[33] 于家畦, 康晓东, 白程程, 等. 一种新的中文电子病历文本检索模型 [J]. 计算机科学, 2022, 49(S1): 32-38. (YU J Q, KANG X D, BAI C C, et al. New text retrieval model of chinese electronic medical records [J]. Computer science, 2022, 49(S1): 32-38.)
[34] CUI Y, CHE W, LIU T, et al. Pre-training with whole word masking for Chinese BERT [J]. IEEE/ACM transactions on audio, speech, and language processing, 2021, 29: 3504-3514.
[35] THECKEDATH D, SEDAMKAR R. Detecting affect states using VGG16, ResNet50 and SE-ResNet50 networks [J]. SN computer science, 2020, 1: 1-7.
[36] CHU Y, YUE X, YU L, et al. Automatic image captioning based on ResNet50 and LSTM with soft attention [J]. Wireless communications and mobile computing, 2020, 2020: 1-7.
[37] 吕学强, 田驰, 张乐, 等. 融合多特征和注意力机制的多模态情感分析模型 [J]. 数据分析与知识发现, 2024, 8(5): 91-101. (LV X Q, TIAN C, ZHANG L, et al. Multi-modal emotion analysis model integrating multi-features and attention mechanism [J]. Data analysis and knowledge discovery, 2024, 8(5): 91-101.)
[38] LIN H, CHENG X, WU X, et al. CAT: cross attention in vision transformer[C]// Proceedings of the 2022 IEEE international conference on multimedia and expo. Taipei: IEEE, 2022, 1-6
Outlines

/