数据集成场景下考虑语义信息的高效实体识别方法

doi:10.13266/j.issn.0252-3116.2022.14.013

图书情报工作 ›› 2022, Vol. 66 ›› Issue (14): 128-136.DOI: 10.13266/j.issn.0252-3116.2022.14.013

数据集成场景下考虑语义信息的高效实体识别方法

宗威, 林松涛, 刘继昶

西安电子科技大学经济与管理学院西安 710126

收稿日期:2021-11-28 修回日期:2022-03-12 出版日期:2022-07-20 发布日期:2022-07-28
作者简介:宗威,副教授,博士,硕士生导师,E-mail:weizong@xidian.edu.cn;林松涛,硕士研究生;刘继昶,本科生。
基金资助:
本文系国家自然科学基金青年基金项目"面向质量-成本权衡分析的多源异构数据集成一致性问题研究"(项目编号:72001164)和陕西省创新能力支撑计划项目"陕西高校科学数据质量评价体系构建与保障策略研究"(项目编号:2022KRM130)研究成果之一。

An Effective Entity Recognition Method Considering Semantic Information in Data Integration Scenario

Zong Wei, Lin Songtao, Liu Jichang

School of Economics and Management, Xidian University, Xi'an 710126

Received:2021-11-28 Revised:2022-03-12 Online:2022-07-20 Published:2022-07-28

摘要/Abstract

摘要： [目的/意义]针对当前数据集成场景下,实体识别时未能充分提取文本语义信息导致识别效果不佳以及传统分块方法无法满足高效识别的问题,提出一种考虑语义信息的高效实体识别方法,以提升实体识别的效果与效率。[方法/过程]以需要集成的两个数据集A、B为例,首先,分别对数据集A和B中的所有记录进行分词、去停用词等数据预处理操作,然后基于数据集A中的每一个词,建立数据集A的倒排索引;其次,计算数据集B中记录的每个词在数据集A中的重要度,依据重要度大小选择关键词代表该条记录;最后将关键词与索引词进行比对,基于Sentence-BERT模型依次计算关键词所对应的记录与索引词包含的所有记录之间的相似程度。将超过阈值的记录判定为对同一实体的描述记录,如此往复直至比对完数据集B中的所有记录。[结果/结论]实验结果表明,本文提出的考虑语义信息的高效实体识别方法在精确率、召回率、稳定性和响应时间等评价指标的表现上均优于传统的实体识别方法,为解决数据集成中的实体识别问题提供了方法指导。

关键词: 实体识别, Sentence-BERT模型, 倒排索引, 数据集成

Abstract: [Purpose/significance] In view of the poor recognition effect caused by the failure to fully extract the text semantic information for entity recognition in the current data integration scene, and the problem that the traditional blocking method can not meet the efficient recognition, an efficient entity recognition method considering semantic information is proposed to improve the effect and efficiency of entity recognition. [Method/process] Taking two data sets A and B that need to be integrated as an example, first, this paper performed word segmentation, removed stop words and conducted other data preprocessing operations on all records in data sets A and B respectively, and then established the inverted index of data set A based on each word in data set A; Secondly, in dataset A, the study calculated the importance of each word recorded in dataset B, and selected a keyword to represent the record according to the importance; Finally, the keyword was compared with the index word, and the similarity between the records corresponding to the keywords and all records contained in the index word was calculated successively based on the Sentence-BERT model. The records exceeding the threshold were determined as the description records of the same entity, and so on until all records in set B were compared. [Result/conclusion] The experimental results show that the efficient entity recognition method considering semantic information proposed in this paper is superior to the traditional entity recognition methods in terms of accuracy, recall, stability and response time, which provides method guidance for solving the entity recognition problem in data integration.

Key words: entity recognition, Sentence-BERT model, inverted index, data integration

中图分类号:

TP391.1

宗威, 林松涛, 刘继昶. 数据集成场景下考虑语义信息的高效实体识别方法[J]. 图书情报工作, 2022, 66(14): 128-136.

Zong Wei, Lin Songtao, Liu Jichang. An Effective Entity Recognition Method Considering Semantic Information in Data Integration Scenario[J]. Library and Information Service, 2022, 66(14): 128-136.

参考文献

[1] SWARTZ N. Gartner warns firms of'dirty data'[J]. Information management journal, 2007, 41(3):6-6.
[2] 王淞,彭煜玮,兰海,等.数据集成方法发展与展望[J].软件学报,2020,31(3):893-908.
[3] 宗威,林松涛,蒿恒,等.基于互异特征向量的ERP重复物料记录识别方法研究[J].信息系统学报,2021(24):92-101.
[4] MARTÍ M, RÓDENAS C. Measurement errors in geographical labor mobility using data linkage:the Spanish case[J]. International journal of social research methodology,2021,24(1):53-64.
[5] TIBBLE H, DI LAW H, et al. The importance of including aliases in data linkage with vulnerable populations[J]. BMC medical research methodology, 2018,18(76):1-5.
[6] 丁东洋,周丽莉.行政记录整合的贝叶斯分层记录链接模型及应用[J].统计与信息论坛,2016,31(7):30-35.
[7] 杨昭,任娟.中文文献题录数据机构名称归一化研究[J].图书情报工作,2020,64(4):95-102.
[8] 林克柔,王昊,龚丽娟,等.融合多特征的中文论文同名学者消歧研究[J].数据分析与知识发现,2021,5(4):90-102.
[9] CHRISTOPHIDES V, EFTHYMIOU V, PALPANAS T, et al. End-to-end entity resolution for big data:a survey[J]. ACM computing surveys, 2021,53(6):1-37.
[10] FELLEGI IP, SUNTER A B. A theory for record linkage[J]. Journal of the American Statistical Association, 1969, 64(328):1183-1210.
[11] HERNÁNDEZ MAURICIO A, STOLFO SALVATORE J. Real-world data is dirty:data cleansing and the merge/purge problem[J]. Data mining and knowledge discovery,1998,2(1):9-37.
[12] XIAO C, WANG W, LIN X, et al. Efficient similarity joins for near-duplicate detection[J]. ACM transactions on database systems, 2011, 36(3):1-41.
[13] 姜华,韩安琪,王美佳,等.基于改进编辑距离的字符串相似度求解算法[J].计算机工程,2014,40(1):222-227.
[14] VATSALAN D, CHRISTEN P, VERYKIOS V S. A taxonomy of privacy-preserving record linkage techniques[J]. Information systems, 2013, 38(6):946-969.
[15] 王瑞琴,杨小明,楼俊钢.词汇语义相关性度量研究[J].情报学报,2016,35(4):389-404.
[16] 牛奉高,张亚宇.基于共现潜在语义向量空间模型的语义核构建[J].情报学报,2017,36(8):834-842.
[17] BENBERNOU S, HUANG X, OUZIRI M. Semantic-based and entity-resolution fusion to enhance quality of big RDF data[J]. IEEE transactions on big data, 2020, 6(2):382-395.
[18] ZHAO H, RAM S. Entity matching across heterogeneous data sources:an approach based on constrained cascade generalization[J]. Data&knowledge engineering, 2008, 66(3):368-381.
[19] 杨萌,聂铁铮,申德荣,等.基于随机森林的实体识别方法[J].集成技术,2018,7(2):57-68.
[20] CUXAC P, LAMIREL J C, BONVALLOT V. Efficient supervised and semi-supervised approaches for affiliations disambiguation[J]. Scientometrics, 2013, 97(1):47-58.
[21] HUANG C, ZHU J, HUANG X, et al. A novel approach for entity resolution in scientific documents using context graphs[J]. Information sciences, 2018(432):431-441.
[22] REIMERS N, GUREVYCH I. Sentence-BERT:sentence embeddings using siamese bert-networks[C]//Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing. Hong Kong:Association for Computational Linguistics, 2019.
[23] HADSELL R, CHOPRA S, YANG L C. Dimensionality reduction by learning an invariant mapping[C]//2006 IEEE computer society conference on computer vision and pattern recognition. New York:IEEE, 2006.
[24] ZONG W, WU F, CHU L K, et al. Identification of approximately duplicate material records in ERP systems[J]. Enterprise information systems, 2017, 11(3):434-451.作者贡献说明:宗威:研究选题与研究思路设计,数据处理,论文修改和审阅;林松涛:数据分析与处理,论文撰写与修改;刘继昶:实验方案设计,实验实施。

数据集成场景下考虑语义信息的高效实体识别方法

An Effective Entity Recognition Method Considering Semantic Information in Data Integration Scenario

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	杜悦, 王东波, 江川, 徐润华, 李斌, 许超, 徐晨飞. 数字人文下的典籍深度学习实体自动识别模型构建及应用研究[J]. 图书情报工作, 2021, 65(3): 100-108.
[2]	刘耀, 张越, 叶璐. 融合篇章结构的文本知识网络构建[J]. 图书情报工作, 2021, 65(21): 118-130.
[3]	王雪, 武俊伟, 陈观群, 李燕琼, 马路. 阿尔茨海默病基因-疾病关联的知识挖掘[J]. 图书情报工作, 2020, 64(13): 120-132.
[4]	赵华茗, 钱力, 余丽. 依存句法特征的科研命名实体识别算法[J]. 图书情报工作, 2020, 64(11): 108-115.
[5]	化柏林, 赵东在, 申泳国. 公共文化服务大数据集成架构设计研究[J]. 图书情报工作, 2020, 64(10): 3-11.
[6]	王若佳, 赵常煜, 王继民. 中文电子病历的分词及实体识别研究[J]. 图书情报工作, 2019, 63(2): 34-42.
[7]	魏巍, 郑杜. 融合统计学习和语义过滤的ADR信号抽取模型构建研究[J]. 图书情报工作, 2018, 62(5): 115-124.
[8]	孙安, 于英香, 罗永刚, 王祺. 序列标注模型中的字粒度特征提取方案研究——以CCKS2017:Task2临床病历命名实体识别任务为例[J]. 图书情报工作, 2018, 62(11): 103-111.
[9]	陈锋, 翟羽佳, 王芳. 基于条件随机场的学术期刊中理论的自动识别方法[J]. 图书情报工作, 2016, 60(2): 122-128.
[10]	侯丽, 李姣, 侯震, 陈松景. 基于混合策略的公众健康领域新词识别方法研究[J]. 图书情报工作, 2015, 59(23): 115-123.
[11]	卓可秋, 童国平, 虞为. 一种基于Spark的论文相似性快速检测方法[J]. 图书情报工作, 2015, 59(11): 134-142.
[12]	Jan Brase, Uwe Rosemann, Irina Sens, 顾立平. 德国国家科技图书馆的发展思考与实践[J]. 图书情报工作, 2013, 57(20): 58-61.
[13]	皇甫晶, 王凌云. 基于规则的纪传体古代汉语文献姓名识别[J]. 图书情报工作, 2013, 57(03): 120-124.
[14]	李益民. 一种基于关键词的大规模DeepWeb信息检索系统[J]. 图书情报工作, 2008, 52(10): 29-32.
[15]	董旻,方曙. Deep Web信息抽取研究[J]. 图书情报工作, 2007, 51(10): 0-0.