收稿日期: 2013-05-28
修回日期: 2013-07-12
网络出版日期: 2013-08-05
基金资助
本文系国家社会科学基金项目“学术文献'意抄'检测研究”(项目编号:12CTQ032)和山东省自然科学基金项目“大规模学术文献并行处理与语义分类研究” (项目编号:ZR2011GL025)研究成果之一。
Literature Similarity Detection Based on Digital Fingerprint
Received date: 2013-05-28
Revised date: 2013-07-12
Online published: 2013-08-05
白如江 , 王晓笛 , 王效岳 . 基于数字指纹的文献相似度检测研究[J]. 图书情报工作, 2013 , 57(15) : 88 -95 . DOI: 10.7536/j.issn.0252-3116.2013.15.014
As a copyright protection technique, digital fingerprint has been a hot research area. This paper proposed a digital fingerprinting algorithm for text based on Chinese words frequency. A frequency list is built through statistics on word frequency and character frequency in a document repository. With this frequency list a digital fingerprint for text of any length can be generated based on the principle for maximum entropy. To get an estimation of the similarity for two texts a Hamming distance can be calculated for the two corresponding digital fingerprint. We build a hash table based on zhwiki-20121129-all-titles corpus and with this table experiment on four core journals. The result shows that normal ways of plagiarism can be detected by this robust fingerprinting algorithm.
[1] Salton G. Cluster search strategies and the optimization of retrieval effectiveness[M]//The SMART Retrieval System. Englewood Cliffs:Prentice Hall, 1971:223-242.
[2] Shivakumar N, Garcia-Molina H. SCAM: A copy detection mechanism for digital documents[C]. 2nd International Conference in Theory and Practice of Digital Libraries. 1995.
[3] Deerwester S, Dumais S T, Furnas G W, et al. Indexing by latent semantic analysis[J]. Journal of the AmerifcanSociety for Information Science, 1990, 41(6): 391-407.
[4] Finkel R A, Zaslavsky A, Monostori K A N, et al. Signature extraction for overlap detection in documents[J]. Australian Computer Science Communications, 2002, 24(1): 59-64.
[5] 刁力力,王丽坤,陆玉昌,等. 计算文本相似度阈值的方法[J]. 清华大学学报(自然科学版),2003(5):108-111.
[6] 金博,史彦军,滕弘飞. 基于语义理解的文本相似度算法[J]. 大连理工大学学报,2005(2):291-297.
[7] 郭武斌,周宽久,苏振魁. 基于词序方法的文本相似度计算模型[J]. 情报学报,2008, 27(6): 857-862.
[8] 吴江宁,刘巧凤. 基于最大公共子图的文本相似度算法研究[J]. 情报学报,2010, 29(5): 785-791.
[9] 黄承慧,印鉴,侯昉. 一种结合词项语义信息和TF-IDF方法的文本相似度量方法[J]. 计算机学报,2011(1):856-858.
[10] 王晋,孙涌,王璁玮. 基于领域本体的文本相似度算法[J]. 苏州大学学报(工科版),2011(2):13-17.
[11] 华秀丽,朱巧明,李培峰. 语义分析与词频统计相结合的中文文本相似度量方法研究[J]. 计算机应用研究,2012(3):833-836.
[12] 杨云,吴亚男,李健. 基于潜在特征词的文本相似度计算方法[J]. 计算机工程与设计,2011(2):572-575.
[13] Rabin M O. Fingerprinting by random polynomials[EB/OL].[2013-07-11].http://www.xmailserver.org/rabin.pdf.
[14] 温泉,孙锬锋,王树勋. 零水印的概念与应用[J]. 电子学报,2003(2): 214-216.
[15] Rivest R.The MD5 message-digest algorithm[EB/OL].[2013-07-11].http://tools.ietf.org/html/rfc1321.
[16] 斯琴,张力,廉德亮. 基于文本特征的文本水印算法[J]. 计算机应用,2009, 29(9): 2348-2350.
[17] 舒娟娟,刘玉玲. 基于词性频率的中文文本零水印算法[J]. 计算机应用,2011(52): 103-105.
[18] Manku G S, Jain A, Das Sarma A. Detecting near-duplicates for Web crawling[M]. New York: ACM, 2007.
[19] Charikar M S. Similarity estimation techniques from rounding algorithms[M]. New York: ACM, 2002.
[20] 李航. 统计学习方法[M].北京:清华大学出版社, 2012.
[21] Hamming R W. Error detecting and error correcting codes[J]. Bell System Technical Journal,1950,26(2): 147-160.
[22] Levenshtein V. Binary codes capable of correcting deletions, insertions[J]. Soviet Physics-Doklady,1966,10(8):707-710.
[23] Maier D. The complexity of some problems on subsequences and supersequences[J]. Journal of ACM, 1978, 2(25): 322-336.
[24] Lowrance R, Wagner R A. An extension of the string-to-string correction problem[J]. Journal of ACM,1975,22(2):177-183.
[25] 董博,郑庆华,宋凯磊,等. 基于多SimHash指纹的近似文本检测[J]. 小型微型计算机系统,2011(11): 2152-2157.
[26] 搜狗实验室[EB/OL].[2013-07-11].http://www.sogou.com/labs/.
[27] Wikimedia downloads[EB/OL].[2013-07-11].http://dumps.wikimedia.org/.
[28] 胡泽文,王效岳,白如江. 国内外文本分类研究计量分析与综述[J]. 图书情报工作,2011,55(6):78-82.
/
| 〈 |
|
〉 |
