图书情报工作

图书情报工作 >

2024 , Vol. 68 >Issue 22: 139 - 152

DOI: https://doi.org/10.13266/j.issn.0252-3116.2024.22.013

知识组织

基于ChatGPT和零样本提示的临床量表文本中结构化项目信息抽取研究

  • 郝洁 ,
  • 莫治强 ,
  • 孙海霞 ,
  • 陈振丽 ,
  • 李姣
展开
  • 1 中国医学科学院/北京协和医学院医学信息研究所 北京 100020;
    2 中国科学技术大学计算机系 合肥 230027
郝洁，助理研究员，博士；莫治强，硕士研究生；陈振丽，硕士研究生；李姣，研究员，博士。

收稿日期: 2024-04-01

  修回日期: 2024-07-18

  网络出版日期: 2024-11-25

基金资助

本文系国家社会科学基金项目“基于知识组织的量表资源语义互联研究”（项目编号：21BTQ069）和中国医学科学院医学与健康科技创新工程项目“医学知识管理与智能化知识服务关键技术研究”（项目编号：2021-I2M-1-056）研究成果之一。

收起

ChatGPT and Zero-Shot Prompt-based Structured Item Information Extraction from Clinical Scale Text

  • Hao Jie ,
  • Mo Zhiqiang ,
  • Sun Haixia ,
  • Chen Zhenli ,
  • Li Jiao
Expand
  • 1 Institute of Medical Information, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing 100020;
    2 Department of Computer Science, University of Science and Technology of China, Hefei 230027

Received date: 2024-04-01

  Revised date: 2024-07-18

  Online published: 2024-11-25

Supported by

This work is supported by the National Social Science Fund of China titled “Semantic Interconnection of Scale Resources Based on Knowledge Organization”(Grant No.21BTQ069) and the Chinese Academy of Medical Sciences Initiative for Innovative Medicine titled “Key Technology of Medical Knowledge Management and Intelligent Knowledge Service”(Grant No.2021-I2M-1-056).

Fold

摘要

[目的/意义] 实现无标注数据情况下使用ChatGPT从自由临床量表文本中抽取结构化的项目信息，高效推进医学量表资源的结构化与智能化。[方法/过程] 定义包含8类属性、兼顾临床量表测量概念结构差异性的项目信息抽取框架，收集59个临床常用心理评定量表文档自建数据集；分类设计零样本提示，调用ChatGPT-3.5和ChatGPT-4官方接口进行实验；多角度分析ChatGPT不同版本在处理不同临床量表文本时的抽取表现和可能影响因素。[结果/结论] 研究结果表明，来源属性抽取表现最佳，Micro-F1和Macro-F1最低也分别达98.90%和97.83%；反应选项、使用说明和计分规则随后；编号和项目指令居中；临床解释最低，Micro-F1和Macro-F1分别为47.73%和45.51%。ChatGPT-4整体表现更优，但部分属性召回率弱于ChatGPT-3.5。量表测量概念层级、维度数、项目数和文本长度的增加会降低模型表现。综上，ChatGPT能够高效辅助医学量表资源的结构化，尤其是在处理简单量表时。

关键词： 医学量表; 文本结构化; 属性抽取; 大语言模型; 零样本学习

本文引用格式

郝洁 , 莫治强 , 孙海霞 , 陈振丽 , 李姣 . 基于ChatGPT和零样本提示的临床量表文本中结构化项目信息抽取研究[J]. 图书情报工作, 2024 , 68(22) : 139 -152 . DOI: 10.13266/j.issn.0252-3116.2024.22.013

Abstract

[Purpose/Significance] This study aims to extract structured item information from free-text clinical scales using ChatGPT without annotations, which efficiently advances the structuring and intellectualization of medical scale resources. [Method/Process] A framework for item information extraction was defined, including eight attribute types and considering the structural differences in clinical scale measurement concepts. A dataset was constructed by collecting 59 commonly used clinical psychometric assessment scale documents. Zero-shot prompt templates were designed based on measurement concept levels, and experiments were conducted using the official ChatGPT-3.5 and ChatGPT-4 interfaces. The extraction performance and possible influencing factors of different ChatGPT versions in processing different clinical scale texts were analyzed from multiple perspectives. [Result/Conclusion] The extraction performance for scale item sources is the best, with Micro-F1 and Macro-F1 scores of at least 98.90% and 97.83%, respectively. This is followed by response options, instructional guidance, and scoring rules, with item numbers and instructions showing moderate performance. Clinical explanations have the lowest performance, with Micro-F1 and Macro-F1 scores of 47.73% and 45.51%, respectively. ChatGPT-4 performs better overall, but the recall rate of some attributes is weaker than that of ChatGPT-3.5. The increase in measurement concept levels, dimensionality, number of items, and text length are found to reduce model performance. In summary, ChatGPT can efficiently assist in the structuring of medical scale resources, especially when dealing with simple scales.

Key words： medical scale; text structuring; attribute extraction; large language model; zero-shot learning

参考文献

[1] 施榕, 郭爱民.全科医生科研方法[M].北京: 人民卫生出版社, 2017: 211-219.(SHI R, GUO A M.Research methods of general practitioners[M].Beijing: People’s Medical Publishing House, 2017: 211-219.)
[2] ZHU M, HONG R H, YANG T, et al.The efficacy of measurement based care for depressive disorders: Systematic review and meta-analysis of randomized controlled trials[J].The journal of clinical psychiatry, 2021, 82(5): 21r14034.
[3] U.S.Food and Drug Administration.Patient-reported outcome measures: use in medical product development to support labeling claims[EB/OL].[2024-07-28].https: //www.fda.gov/regulatory-information/search-fda-guidance-documents
[4] DEVELLIS R F.Scale development: Theory and applications[M].4th ed.Los Angeles: SAGE, 2017.
[5] CELLA D, YOUNT S, ROTHROCK N, et al.The patient-reported outcomes measurement information system (PROMIS): progress of an NIH roadmap cooperative group during its first two years[J].Medical care, 2007, 45(5 Suppl 1): S3-S11.
[6] NLM.NIH CDE repository[EB/OL].[2024-08-02].https: //cde.nlm.nih.gov/form/search.
[7] ZHAO W X, ZHOU K, LI J Y, et al.A survey of large language models[J].arXiv preprint arXiv: 2303.18223, 2023.
[8] GOEL A, GUETA A, GILON O, et al.LLMs accelerate annotation for medical information extraction[C]//Machine learning for health (ML4H).New York: PMLR, 2023: 82-100.
[9] WAN Z, CHENG F, MAO Z, et al.GPT-RE: In-context learning for relation extraction using large language models[J].arXiv preprint arXiv: 2305.02105, 2023.
[10] 杨冬菊, 黄俊涛.基于大语言模型的中文科技文献标注方法[J/OL].计算机工程: 1-7[2024-03-30].https: //doi.org/10.19678/j.issn.1000-3428.0068400.(YANG D G, HUANG J T.Workshop on enterprise knowledge graphs using large language models[J/OL].Computer engineering, 1-7[2024-03-30].https: //doi.org/10.19678/j.issn.1000-3428.0068400.)
[11] KARAN S, TAO T, JURAJ G, et al.Towards expert-level medical question answering with large language models[J].arXiv preprint arXiv: 2305.09617, 2023.
[12] FREY J, MEYER L P, ARNDT N, et al.Benchmarking the abilities of large language models for RDF knowledge graph creation and comprehension: how well do LLMs speak turtle?[J].arXiv preprint arXiv: 2309.17122, 2023.
[13] KUMAR A H S.Analysis of ChatGPT tool to assess the potential of its utility for academic writing in biomedical domain[J].Biology, engineering, medicine and science reports, 2023, 9(1): 24-30.
[14] MANNING C D.Human language understanding & reasoning [J].Daedalus, 2022, 151(2): 127-138.
[15] TOUVRON H, LAVRIL T, IZACARD G, et al.LLaMA: open and efficient foundation language models[J].arXiv preprint arXiv: 2302.13971, 2023.
[16] LIU P, YUAN W, FU J, et al.Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing[J].ACM computing surveys, 2023, 55(9): 1-35.
[17] HAN X, ZHANG Z, DING N, et al.Pre-trained models: past, present and future[J].arXiv preprint arXiv: 2106.07139, 2021.
[18] CHENG P, Xi Y, AOKUN C, et al.A study of generative large language model for medical research and healthcare[J].arXiv preprint arXiv: 2305.13523, 2023.
[19] 马武仁, 弓孟春, 戴辉, 等.以ChatGPT为代表的大语言模型在临床医学中的应用综述[J].医学信息学杂志, 2023, 44(7): 9-17.(MA W R, GONG M C, DAI H, et al.A comprehensive review of the applications of large language models in clinical medicine with ChatGPT as a representative[J].Journal of medical informatics, 2023, 44(7): 9-17.)
[20] MONICA A, STEFAN H, HUNTER L, et al.Large language models are few-shot clinical information extractors[C]//Proceedings of the 2022 conference on empirical methods in natural language processing.Abu Dhabi: ACL, 2022: 1998-2022.
[21] PENG C, YANG X, CHEN A, et al.A study of generative large language model for medical research and healthcare[J].NPJ digital medicine, 2023, 6(1): 210.
[22] AGRAWAL M, HEGSELMANN S, LANG H, et al.Large language models are few-shot clinical information extractors[J].arXiv preprint arXiv: 2205.12689, 2022.
[23] MIWA M, BANSAL M.End-to-end relation extraction using LSTMs on sequences and tree structures[J].arXiv preprint arXiv: 1601.00770, 2016.
[24] BEKOULIS G, DELEU J, DEMEESTER T, et al.Joint entity recognition and relation extraction as a multi-head selection problem[J].Expert systems with applications, 2018, 114: 34-45.
[25] LUAN Y, HE L, OSTENDORF M, et al.Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction[J].arXiv preprint arXiv: 1808.09602, 2018.
[26] 杨波, 孙晓虎, 党佳怡, 等.面向医疗问答系统的大语言模型命名实体识别方法[J].计算机科学与探索, 2023, 17(10): 2389.(YANG B, SUN X H, DANG J Y, et al.Named entity recognition method of large language model for medical question answering system[J].Journal of frontiers of computer science and technology, 2023, 17(10): 2389.)
[27] SUN X, LI X, LI J, ET A L.Text classification via large language models[J].arXiv preprint arXiv: 2305.08377, 2023.
[28] ONG J, KEDIA N, HARIHAR S, et al.Applying large language model artificial intelligence for retina international classification of diseases (ICD) coding[J].Journal of medical artificial intelligence, 2023, 6: 21.
[29] WU L I, LI G.Zero-shot construction of Chinese medical knowledge graph with ChatGPT[C]//2023 IEEE international conference on medical artificial intelligence (MedAI).Beijing: IEEE, 2023: 278-283.
[30] ZHU Y, WANG X, CHEN J, et al.LLMs for knowledge graph construction and reasoning: Recent capabilities and future opportunities[J].arXiv preprint arXiv: 2305.13168, 2023.
[31] ZHU W, WANG X, CHEN M, et al.Overview of the PromptCBLUE shared tTask in CHIP2023[J].arXiv preprint arXiv: 2312.17522, 2023.
[32] LI B, FANG G, YANG Y, et al.Evaluating ChatGPT’s information extraction capabilities: an assessment of performance, explainability, calibration, and faithfulness[J].arXiv preprint arXiv: 2304.11633, 2023.
[33] WEI X, CUI X, CHENG N, et al.Zero-shot information extraction via chatting with ChatGPT[J].arXiv preprint arXiv: 2302.10205, 2023.
[34] MIALON, G.DESSÌ R, LOMELI M, et al.Augmented language models: A survey[J].arXiv preprint arXiv: 2302.07842, 2023.
[35] DUNN A, DAGDELEN J, WALKER N, et al.Structured information extraction from complex scientific text with fine-tuned large language models[J].arXiv preprint arXiv: 2212.05238, 2022.
[36] ZHENG C, GUO C Y.A pattern-first pipeline approach for entity and relation extraction[J].Neurocomputing, 2022(494): 182-191.
[37] XUE L L, ZHANG D, DONG Y X, et al.AutoRE: Document-level relation extraction with large language models[J].arXiv preprint arXiv: 2403.14888, 2024.
[38] 孙海霞, 郝洁, 郭臻, 等.基于知识元的细粒度医学量表文档知识表示框架构建[J].数字图书馆论坛, 2023, 19(12): 86-98.(SUN H X, HAO J, GUO Z, et al.Construction of a fine-grained knowledge element-based framework for knowledge representation in medical scale documents[J].Digital library forum, 2023, 19(12): 86-98.)
[39] 戴晓阳.常用心理评估量表手册[M].修订版.北京: 人民军医出版社, 2015: 150-154.(DAI X Y.Handbook of commonly used psychological assessment scales[M].Revised ed.Beijing: People’s Military Medical Publishing House, 2015: 150-154.)
[40] HUANG J W, YANG D H M, RONG R C, et al.A critical assessment of using ChatGPT for extracting structured data from clinical notes[J].NPJ digital medicine, 2024(7): 106.
[41] LANDIS J R, KOCH G G.The measurement of observer agreement for categorical data[J].Biometrics, 33(1): 159-174.
[42] NG A, FULFORD L.ChatGPT prompt engineering for developers[EB/OL].[2024-08-03].https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/.
[43] YUAN W, NEUBIG G, LIU P.Bartscore: Evaluating generated text as text generation[C]//Proceedings of the 34th international conference on neural information processing systems.New York: Curran Associates, 2021.
[44] CHEN J, LIU L, RUAN S, et al.Are different versions of ChatGPT’s ability comparable to the clinical diagnosis presented in case reports? a descriptive study[J].Journal of multidisciplinary healthcare, 2023, 16: 3825-3831.
Options
文章导航

/

〈 〉