RESEARCH PAPERS

Automatic Sentence Segmentation and Automatic Punctuation for Ancient Chinese Texts Based on Generative Large Language Model

  • Xia Tian ,
  • Yu Kai ,
  • Yu Qianrong ,
  • Peng Xinran ,
  • Zhao Qun ,
  • Yang Menghui
Expand
  • 1 School of Information Resource Management, Renmin University of China, Beijing 100872;
    2 Midu Technology Co., Ltd., Shanghai 201204
Xia Tian,associate professor,PhD;Yu Kai,master;Yu Qianrong,master candidate;Peng Xinran,master candidate;Zhao Qun,doctoral candidate;Yang Menghui,professor,PhD,doctoral supervisor,corresponding author,E-mail:yangmenghui@ruc.edu.cn.

Received date: 2024-06-03

  Revised date: 2024-08-17

  Online published: 2025-03-07

Supported by

This work is supported by the National Social Science Fund of China project titled “Research on Knowledge Discovery and Intelligent Services Based on the Full Content of Books” (Grant No. 22BTQ068).

Abstract

[Purpose/Significance] This study aims to improve automatic sentence segmentation and punctuation in ancient Chinese texts using generative large language models, overcoming the limitations of traditional sequence labeling models that require specialized label design and annotated data construction. [Method/Process] In order to increase the number of learnable samples, a sliding window strategy was used to segment training data. Additionally, MinHash and Locality-Sensitive Hashing (LSH) were employed to provide reference samples for unpunctuated texts, and constraints were applied to the decoding process of the large language model. The Xunzi ancient literature large language model served as the base model. Fine-tuning was performed using Low-Rank Adaptation (LoRA) to align the model with the task of ancient Chinese automatic punctuation, enabling the generation of target texts with punctuation from unpunctuated texts. [Result/Conclusion] On two benchmark datasets released by EvaHan 2024, the F1 scores of automatic sentence segmentation are 88.47% and 92.48%, and the F1 scores of automatic punctuation are 75.29% and 80.01%, respectively. These results significantly outperform the Xunzi ancient large language model and ChatGPT 3.5, indicating that generative large language models are a feasible approach for addressing the tasks of sentence segmentation and punctuation in ancient Chinese texts.

Cite this article

Xia Tian , Yu Kai , Yu Qianrong , Peng Xinran , Zhao Qun , Yang Menghui . Automatic Sentence Segmentation and Automatic Punctuation for Ancient Chinese Texts Based on Generative Large Language Model[J]. Library and Information Service, 2025 , 69(5) : 59 -70 . DOI: 10.13266/j.issn.0252-3116.2025.05.006

References

[1] 袁义国, 李斌, 冯敏萱,等.基于深度学习的古籍文本自动断句与标点一体化研究[J]. 图书情报工作, 2022, 66(22): 134-141. (YUAN Y G, LI B, FENG M X, et al. A joint model of automatic sentence segmentation and punctuation for ancient classical texts based on deep learning[J]. Library and information service, 2022(22): 134-141.)
[2] 国家标准化管理委员会. GB/T 15834-2011标点符号用法[S]. 北京: 中国标准出版社, 2021. (National Standardization Administration. GB/T 15834-2011. General rules for punctuation [S]. Beijing: Standardization Press of China, 2021.)
[3] 赵连振, 张逸勤, 刘江峰, 等. 面向数字人文的先秦两汉典籍自动标点研究——以SikuBERT预训练模型为例[J]. 图书馆论坛, 2022, 42(12): 120-128, 137. (ZHAO L Z, ZHANG Y Q, LIU J F, et al. Study on Automatic punctuation of ancient Chinese classics of pre-Qin and Han dynasties in the context of digital humanities: taking Sikubert pre-training model for example [J]. Library tribune, 2022, 42(12): 120-128, 137.)
[4] 南京农业大学.荀子古籍大语言模型[EB/OL]. [2024-05-30]. https://github.com/Xunzi-LLM-of-Chinese-classics/XunziALLM.
[5] LI B, CHANG B L, XU Z X, et al. Overview of EvaHan2024: the first international evaluation on ancient Chinese sentence segmentation and punctuation[C]// SPRUGNOLI R, PASSAROTTI M. Proceedings of the third workshop on language technologies for historical and ancient languages, pages. Torino: ACL, 2024: 229-236.
[6] 黄建年, 侯汉清. 农业古籍断句标点模式研究[J]. 中文信息学报, 2008, 22(4): 31-38.(HUANG J N, HOU H Q. On sentence segmentation and punctuation model for ancient books on agriculture [J]. Journal of Chinese information processing, 2008, 22(4): 31-38.)
[7] 陈天莹, 陈蓉, 潘璐璐, 等. 基于前后文n-gram模型的古汉语句子切分[J]. 计算机工程, 2007(3): 192-193, 196. (CHEN T R, CHEN R, PAN L L, et al. Archaic Chinese punctuating sentences based on context n-gram model[J]. Computer engineering, 2007(3): 192-193, 196.)
[8] 黄瀚萱. 以序列标注方法解决古汉语断句问题[D]. 台湾: 台湾交通大学, 2008. (HUANG H H. Classical Chinese sentence division by sequence labeling approaches[D]. Taiwan: “National” Chiao Tung University, 2008.)
[9] 张开旭, 夏云庆, 宇航. 基于条件随机场的古汉语自动断句与标点方法[J]. 清华大学学报(自然科学版), 2009, 49(10): 1733-1736. (ZHANG K X, XIA Y Q, YU H. CRF-based approach to sentence segmentation and punctuation for ancient Chinese prose [J]. Journal of Tsinghua University (science and technology), 2009, 49(10): 1733-1736.)
[10] 张合, 王晓东, 杨建宇, 等. 一种基于层叠CRF的古文断句与句读标记方法[J]. 计算机应用研究, 2009, 26(9): 3326-3329. (ZHANG H, WANG X D, YANG J Y, et al. Method of sentence segmentation and punctuating for ancient Chinese literatures based on cascaded CRF [J]. Application research of computers, 2009, 26(9): 3326-3329.)
[11] HUANG H H, SUN C T, CHEN H H. Classical Chinese sentence segmentation[C]//SUN L. CIPS-SIGHAN joint conference on Chinese language processing. Stroudsburg: ACL, 2010: 15-22.
[12] 许京奕. 古汉语文本自动句读研究[D]. 北京: 北京大学, 2011. (XU J Y. Sentence segmentation in classical Chinese texts[D]. Beijing: Peking University, 2011.)
[13] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[C]//BURSTEIN J, DORAN C, SOLORIO T. Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: human language technologies. Stroudsburg: ACL, 2019: 4171-4186.
[14] 王博立, 史晓东, 苏劲松. 一种基于循环神经网络的古文断句方法[J]. 北京大学学报(自然科学版), 2017, 53(2): 255-261. (WANG B L, SHI X D, SU J S. A sentence segmentation method for ancient Chinese texts based on recurrent neural network [J]. Acta scientiarum naturalium universitatis pekinensis, 2017, 53(2): 255-261.)
[15] HAN X, WANG H S, ZHANG S Q, et al. Sentence segmentation for classical Chinese based on LSTM with radical embedding[J]. The journal of China Universities of Posts and Telecommunications, 2019, 26(2): 1-8.
[16] WANG H B, WEI H B, GUO J Y, et al. Ancient Chinese sentence segmentation based on bidirectional LSTM[J]. Journal of advanced computational intelligence and intelligent informatics, 2019, 23(4): 719-725.
[17] 程宁, 李斌, 葛四嘉, 等. 基于BiLSTM-CRF的古汉语自动断句与词法分析一体化研究[J]. 中文信息学报, 2020, 34(4): 1-9. (CHENG N, LI B, GE S J, et al. A joint model of automatic sentence segmentation and lexical analysis for ancient Chinese based on BiLSTM-CRF Model [J]. Journal of Chinese information processing, 2020, 34(4): 1-9.)
[18] 俞敬松, 魏一, 张永伟. 基于BERT的古文断句研究与应用[J]. 中文信息学报, 2019, 33(11): 57-63. (YU J S, WEI Y, ZHANG Y W. Automatic ancient Chinese texts segmentation based on BERT [J]. Journal of Chinese information processing, 2019, 33(11): 57-63.)
[19] 王倩, 王东波, 李斌, 等. 面向海量典籍文本的深度学习自动断句与标点平台构建研究[J]. 数据分析与知识发现, 2021, 5(3): 25-34. (WANG Q, WANG D B, LI B, et al. Deep learning based automatic sentence segmentation and punctuation model for massive classical Chinese literature [J]. Data analysis and knowledge discovery, 2021, 5(3): 25-34.)
[20] 胡韧奋, 李绅, 诸雨辰. 基于深层语言模型的古汉语知识表示及自动断句研究[J]. 中文信息学报, 2021, 35(4): 8-15. (HU R F, LI S, ZHU Y C. Knowledge representation and sentence segmentation of ancient Chinese based on deep language models [J]. Journal of Chinese information processing, 2021, 35(4): 8-15.)
[21] 李佩琪, 王昊, 任秋彤, 等. 融合结构特性的语义增强式古籍句读识别方法研究[J]. 情报学报, 2023, 42(2): 150-163. (LI P Q, WANG H, REN Q T, et al. Study of antiquarian punctuation recognition methods incorporating semantic enhancement with structural properties [J]. Journal of the China Society for Scientific and Technical Information, 2023, 42(2): 150-163.)
[22] 张景素, 魏明珠.基于加权多策略选样的古文断句模型研究——以古籍《宋史》为例[J]. 情报科学, 2022, 40(10): 164-170. (ZHANG J S, WEI M Z. Sentence segmentation model of ancient books text based on weighted multi-strategy sampling: taking history of Song Dynasty as an example [J]. Information science, 2022, 40(10): 164-170.)
[23] 唐雪梅, 苏祺, 王军, 等. 基于预训练语言模型的繁体古文自动句读研究[J]. 中文信息学报, 2023, 37(8): 159-168. (TANG X M, SU Q, WANG J, et al. Automatic traditional ancient Chinese texts segmentation and punctuation based on pre-trained language model [J]. Journal of Chinese information processing, 2023, 37(8): 159-168.)
[24] 王东波, 刘畅, 朱子赫, 等. SikuBERT与SikuRoBERTa: 面向数字人文的《四库全书》预训练模型构建及应用研究[J]. 图书馆论坛, 2022, 42(6): 31-43. (WANG D B, LIU C, ZHU Z H, et al. Construction and application of pre-trained models of Siku Quanshu in orientation to digital humanities [J]. Library tribune, 2022, 42(6): 31-43.)
[25] HU E J, SHEN Y L, WALLIS P, et al. LoRA: low-rank adaptation of large language models[C]//Proceedings of the 10th international conference on learning representations. Online: ICLR, 2022.
[26] LEWIS P, PEREZ E, PIKTUS A, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks[C]//LAROCHELLE H, RANZATO M, HADSELL R, et al. Advances in Neural Information Processing Systems 33. New York: NeurIPS, 2020: 9459-9474.
[27] BAWA M, CONDIE T, GANESAN P. LSH Forest: self-tuning indexes for similarity search[C]// ELLIS A, HAGINO T. Proceedings of the 14th international conference on World Wide Web. New York: ACM, 2005: 651-660.
[28] BAI J Z, BAI S, CHU Y F, et al. Qwen technical report[EB/OL]. [2024-05-30]. https://arxiv.org/abs/2309.16609.
Outlines

/

〈 〉