[目的/意义] 在科技文献中，插图与文本作为两种常见且互补的信息呈现方式，共同构成知识传播的重要组成部分。通过识别文献中的插图及其关联文本，实现多模态信息的整合与利用，进而提升科技文献的知识挖掘与检索效率，对科技文献资源建设以及多模态知识服务发展具有重要意义。[方法/过程] 提出一个针对科技文献中插图知识单元构建向量库的方法，从插图知识单元识别的准确性、完整性和关联性3个维度设计模型；在此基础上，设计向量表征和存储方式进一步构建插图向量库；最终，基于所构建的论文插图向量库，搭建论文插图检索系统，实现科技文献的多模态知识服务。[结果/结论] 基于提出的方法，插图知识单元识别的F1值为84.1%，论文插图和关联文本片段识别的F1值分别为99.5%、89.0%，并成功抽取化学化工领域的120万篇科技论文，构建百万级论文插图检索系统，为多模态知识挖掘及知识服务提供基础支撑。
[Purpose/Significance] In scientific literature, figures and text, as two common and complementary forms of information presentation, together constitute a crucial component of knowledge dissemination. By recognizing figures and their associated textual content within literature, the integration and utilization of multimodal information can be achieved, thereby enhancing the efficiency of knowledge extraction and retrieval from scientific documents. It has significant implications for the construction of scientific literature resources and the development of multimodal knowledge services. [Method/Process] This paper proposed a method for constructing a vector database of figure knowledge units in scientific literature. The model was designed to evaluate figure knowledge units based on three key dimensions: accuracy, completeness, and relevance. Building upon this model, a vector representation and storage mechanism were developed to further construct the figure vector database. Finally, an end-to-end figure retrieval system was built upon the constructed database to enable multimodal knowledge services in scientific literature. [Result/Conclusion] Based on the method proposed in this paper, the recognition of figure knowledge units achieves an F1 score of 84.1%. The recognition of figures and their associated text fragments achieves F1 scores of 99.5% and 89.0% respectively. Additionally, it effectively extracts data from 1.2 million chemical literature papers. It has led to the development of a large-scale figure retrieval system, providing foundational support for multimodal knowledge extraction and knowledge services.
[1] 徐雷, 秦翠玉, 李娇. 科技文献数据化及组织呈现路径研究[J]. 中国图书馆学报, 2022, 48(3): 25-42. (Xu L, Qin C Y, Li J. Datafication, organization and manifestation of scientific literature [J]. Journal of library science in China, 2022, 48(3): 25-42.)
[2] CLARK C, DIVVALA S. PDFFigures 2.0: mining figures from research papers[C]//Proceedings of the 16th ACM/IEEE-CS joint conference on digital libraries. New York: Association for Computing Machinery, 2016: 143-152.
[3] LEE P-S, WEST J D, HOWE B. Viziometrics: analyzing visual information in the scientific literature [J]. IEEE transactions on big data, 2016, 4(1): 117-129.
[4] GUO W Z, WANG J W, WANG S P. Deep multimodal representation learning: a survey [J]. IEEE access, 2019, 7: 63373-63394.
[5] 任林涛. PDF格式中文科技论文的有效信息提取方法及分类研究[D]. 长春: 吉林大学, 2011. (Ren L T. Extraction effective information and calssification of PDF format of scientific papers in Chinese [D]. Changchun: Jilin University, 2011.)
[6] 胡志刚, 田文灿, 孙太安, 等. 科技论文中学术信息的提取方法综述[J]. 数字图书馆论坛, 2017(10): 39-47. (HU Z G, TIAN W C, SUN T A, et al. A method review on academic information extracting from scientific papers [J]. Digital library forum, 2017(10): 39-47.)
[7] LI P, JIANG X, KAMBHAMETTU C, et al. Compound image segmentation of published biomedical figures [J]. Bioinformatics, 2018, 34(7): 1192-1199.
[8] CHAKI J. An automatic system for extracting figure-caption pair from medical documents: a six-fold approach [J]. PeerJ computer science, 2023(9): e1452.
[9] NILANJAN D, SUREKHA B, AMIRA S A, et al. Machine learning in bio-signal analysis and diagnostic imaging[M]. Berlin: Springer, 2019: 247-272.
[10] PRACZYK P, NOGUERAS-ISO J. Automatic extraction of figures from scientific publications in high-energy physics [J]. Information technology and libraries, 2013(32): 25-52.
[11] LOPEZ L D, YU J, ARIGHI C N, et al. An automatic system for extracting figures and captions in biomedical pdf documents[C]//IEEE international conference on bioinformatics and biomedicine. Atlanta: IEEE Computer Society, 2011: 578-581.
[12] CHOUDHURY S R, TUAROB S, MITRA P, et al. A figure search engine architecture for a chemistry digital library[C]//Proceedings of the 13th ACM/IEEE-CS joint conference on digital libraries. New York: Association for Computing Machinery, 2013: 369-370.
[13] SIEGEL N, LOURIE N, POWER R, et al. Extracting scientific figures with distantly supervised neural networks[C]//Proceedings of the 18th ACM/IEEE on joint conference on digital libraries. New York: Association for Computing Machinery, 2018: 223-232.
[14] CLARK C, DIVVALA S K. Looking beyond text: extracting figures, tables and captions from computer science papers[C]//AAAI workshop on scholarly big data. Austin: AAAI Technical Report, 2015.
[15] LI P, JIANG X, SHATKAY H. Figure and caption extraction from biomedical documents [J]. Bioinformatics, 2019, 35(21): 4381-4388.
[16] DARMANOVIĆ F, HANBURY A, ZLABINGER M. SCI-3000: a dataset for figure, table and caption extraction from scientific PDFs[C]//ICDAR 2023: 17th international conference. San José: Springer, 2023: 234-251.
[17] BISHOP A P. Document structure and digital libraries: how researchers mobilize information in journal articles [J]. Information processing & management, 1999, 35(3): 255-279.
[18] SANDUSKY R J, TENOPIR C, CASADO M M. Figure and table retrieval from scholarly journal articles: user needs for teaching and research[C]//Proceedings of the American Society for Information Science and Technology. Columbus: ASIS&T, 2008, 44: 1-13.
[19] National Library of Medicine. Open-i [EB/OL]. [2025-02-25].https://openi.nlm.nih.gov.
[20] National Library of Medicine. PMC images [EB/OL]. [2025-02-25]. https://pmc.ncbi.nlm.nih.gov/.
[21] Allen Institute for Artificial Intelligence. Semantic scholar [EB/OL]. [2025-02-25]. https://www.semanticscholar.org/.
[22] 中国知网. CNKI学术知识图片库[EB/OL]. [2025-02-25].https://image.cnki.net. (China National Knowledge Infrastructure. CNKI academic knowledge figure library [EB/OL]. [2025-02-25]. https://image.cnki.net.)
[23] 索传军, 盖双双. 知识元的内涵、结构与描述模型研究[J]. 中国图书馆学报, 2018, 44(4): 54-72. (SUO C J, GAI S S. The connotation, structure and description model of knowledge unit [J]. Journal of library science in China, 2018, 44(4): 54-72.)
[24] 戎军涛. 学术文献内容知识元语义描述模型研究[J]. 情报科学, 2019, 37(7): 30-35. (RONG J T. Semantic description model of academic literature content based on knowledge element [J]. Information science, 2019, 37(7): 30-35.)
[25] 温有奎, 焦玉英. 基于知识元的知识发现[M]. 西安: 西安电子科技大学出版社, 2011. (WEN Y K, JIAO Y Y. Knowledge discovery based on knowledge elements [M]. Xi’an: Xidian University Press, 2011.)
[26] 唐晓波, 陈俭静, 周禾深, 等. 基于知识元的科学-技术知识关联指标与测度方法研究[J]. 数字图书馆论坛, 2024, 20(2): 58-69. (TANG X B, CHEN J J, ZHOU H S, et al. Indicators and measurement method of science-technology knowledge linkage based on knowledge elements [J]. Digital library forum, 2024, 20(2): 58-69.)
[27] 石湘, 刘萍. 基于知识元语义描述模型的领域知识抽取与表示研究——以信息检索领域为例[J]. 数据分析与知识发现, 2021, 5(4): 123-33. (SHI X, LIU P. Extraction and representation of domain knowledge with semantic description model and knowledge elements: case study of information retrieval [J]. Data analysis and knowledge discovery, 2021, 5(4): 123-133.)
[28] 索传军, 戎军涛. 知识元理论研究述评[J]. 图书情报工作, 2021, 65(11): 133-142. (SUO C J, RONG J T. Review on the theory of knowledge element [J]. Library and information service, 2021, 65(11): 133-142.)
[29] ZHANG B, ZHANG P, DONG X, et al. Long-CLIP: unlocking the long-text capability of CLIP [J]. arXiv preprint, 2024, arXiv:240315378.
[30] TAIPALUS T. Vector database management systems: fundamental concepts, use-cases, and current challenges [J]. Cognitive systems research, 2024(85): 101216.
