KNOWLEDGE ORGANIZATION

ChatGPT and Zero-Shot Prompt-based Structured Item Information Extraction from Clinical Scale Text

  • Hao Jie ,
  • Mo Zhiqiang ,
  • Sun Haixia ,
  • Chen Zhenli ,
  • Li Jiao
Expand
  • 1 Institute of Medical Information, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing 100020;
    2 Department of Computer Science, University of Science and Technology of China, Hefei 230027

Received date: 2024-04-01

  Revised date: 2024-07-18

  Online published: 2024-11-25

Supported by

This work is supported by the National Social Science Fund of China titled “Semantic Interconnection of Scale Resources Based on Knowledge Organization”(Grant No.21BTQ069) and the Chinese Academy of Medical Sciences Initiative for Innovative Medicine titled “Key Technology of Medical Knowledge Management and Intelligent Knowledge Service”(Grant No.2021-I2M-1-056).

Abstract

[Purpose/Significance] This study aims to extract structured item information from free-text clinical scales using ChatGPT without annotations, which efficiently advances the structuring and intellectualization of medical scale resources. [Method/Process] A framework for item information extraction was defined, including eight attribute types and considering the structural differences in clinical scale measurement concepts. A dataset was constructed by collecting 59 commonly used clinical psychometric assessment scale documents. Zero-shot prompt templates were designed based on measurement concept levels, and experiments were conducted using the official ChatGPT-3.5 and ChatGPT-4 interfaces. The extraction performance and possible influencing factors of different ChatGPT versions in processing different clinical scale texts were analyzed from multiple perspectives. [Result/Conclusion] The extraction performance for scale item sources is the best, with Micro-F1 and Macro-F1 scores of at least 98.90% and 97.83%, respectively. This is followed by response options, instructional guidance, and scoring rules, with item numbers and instructions showing moderate performance. Clinical explanations have the lowest performance, with Micro-F1 and Macro-F1 scores of 47.73% and 45.51%, respectively. ChatGPT-4 performs better overall, but the recall rate of some attributes is weaker than that of ChatGPT-3.5. The increase in measurement concept levels, dimensionality, number of items, and text length are found to reduce model performance. In summary, ChatGPT can efficiently assist in the structuring of medical scale resources, especially when dealing with simple scales.

Cite this article

Hao Jie , Mo Zhiqiang , Sun Haixia , Chen Zhenli , Li Jiao . ChatGPT and Zero-Shot Prompt-based Structured Item Information Extraction from Clinical Scale Text[J]. Library and Information Service, 2024 , 68(22) : 139 -152 . DOI: 10.13266/j.issn.0252-3116.2024.22.013

References

[1] 施榕, 郭爱民.全科医生科研方法[M].北京: 人民卫生出版社, 2017: 211-219.(SHI R, GUO A M.Research methods of general practitioners[M].Beijing: People’s Medical Publishing House, 2017: 211-219.)
[2] ZHU M, HONG R H, YANG T, et al.The efficacy of measurement based care for depressive disorders: Systematic review and meta-analysis of randomized controlled trials[J].The journal of clinical psychiatry, 2021, 82(5): 21r14034.
[3] U.S.Food and Drug Administration.Patient-reported outcome measures: use in medical product development to support labeling claims[EB/OL].[2024-07-28].https: //www.fda.gov/regulatory-information/search-fda-guidance-documents
[4] DEVELLIS R F.Scale development: Theory and applications[M].4th ed.Los Angeles: SAGE, 2017.
[5] CELLA D, YOUNT S, ROTHROCK N, et al.The patient-reported outcomes measurement information system (PROMIS): progress of an NIH roadmap cooperative group during its first two years[J].Medical care, 2007, 45(5 Suppl 1): S3-S11.
[6] NLM.NIH CDE repository[EB/OL].[2024-08-02].https: //cde.nlm.nih.gov/form/search.
[7] ZHAO W X, ZHOU K, LI J Y, et al.A survey of large language models[J].arXiv preprint arXiv: 2303.18223, 2023.
[8] GOEL A, GUETA A, GILON O, et al.LLMs accelerate annotation for medical information extraction[C]//Machine learning for health (ML4H).New York: PMLR, 2023: 82-100.
[9] WAN Z, CHENG F, MAO Z, et al.GPT-RE: In-context learning for relation extraction using large language models[J].arXiv preprint arXiv: 2305.02105, 2023.
[10] 杨冬菊, 黄俊涛.基于大语言模型的中文科技文献标注方法[J/OL].计算机工程: 1-7[2024-03-30].https: //doi.org/10.19678/j.issn.1000-3428.0068400.(YANG D G, HUANG J T.Workshop on enterprise knowledge graphs using large language models[J/OL].Computer engineering, 1-7[2024-03-30].https: //doi.org/10.19678/j.issn.1000-3428.0068400.)
[11] KARAN S, TAO T, JURAJ G, et al.Towards expert-level medical question answering with large language models[J].arXiv preprint arXiv: 2305.09617, 2023.
[12] FREY J, MEYER L P, ARNDT N, et al.Benchmarking the abilities of large language models for RDF knowledge graph creation and comprehension: how well do LLMs speak turtle?[J].arXiv preprint arXiv: 2309.17122, 2023.
[13] KUMAR A H S.Analysis of ChatGPT tool to assess the potential of its utility for academic writing in biomedical domain[J].Biology, engineering, medicine and science reports, 2023, 9(1): 24-30.
[14] MANNING C D.Human language understanding & reasoning [J].Daedalus, 2022, 151(2): 127-138.
[15] TOUVRON H, LAVRIL T, IZACARD G, et al.LLaMA: open and efficient foundation language models[J].arXiv preprint arXiv: 2302.13971, 2023.
[16] LIU P, YUAN W, FU J, et al.Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing[J].ACM computing surveys, 2023, 55(9): 1-35.
[17] HAN X, ZHANG Z, DING N, et al.Pre-trained models: past, present and future[J].arXiv preprint arXiv: 2106.07139, 2021.
[18] CHENG P, Xi Y, AOKUN C, et al.A study of generative large language model for medical research and healthcare[J].arXiv preprint arXiv: 2305.13523, 2023.
[19] 马武仁, 弓孟春, 戴辉, 等.以ChatGPT为代表的大语言模型在临床医学中的应用综述[J].医学信息学杂志, 2023, 44(7): 9-17.(MA W R, GONG M C, DAI H, et al.A comprehensive review of the applications of large language models in clinical medicine with ChatGPT as a representative[J].Journal of medical informatics, 2023, 44(7): 9-17.)
[20] MONICA A, STEFAN H, HUNTER L, et al.Large language models are few-shot clinical information extractors[C]//Proceedings of the 2022 conference on empirical methods in natural language processing.Abu Dhabi: ACL, 2022: 1998-2022.
[21] PENG C, YANG X, CHEN A, et al.A study of generative large language model for medical research and healthcare[J].NPJ digital medicine, 2023, 6(1): 210.
[22] AGRAWAL M, HEGSELMANN S, LANG H, et al.Large language models are few-shot clinical information extractors[J].arXiv preprint arXiv: 2205.12689, 2022.
[23] MIWA M, BANSAL M.End-to-end relation extraction using LSTMs on sequences and tree structures[J].arXiv preprint arXiv: 1601.00770, 2016.
[24] BEKOULIS G, DELEU J, DEMEESTER T, et al.Joint entity recognition and relation extraction as a multi-head selection problem[J].Expert systems with applications, 2018, 114: 34-45.
[25] LUAN Y, HE L, OSTENDORF M, et al.Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction[J].arXiv preprint arXiv: 1808.09602, 2018.
[26] 杨波, 孙晓虎, 党佳怡, 等.面向医疗问答系统的大语言模型命名实体识别方法[J].计算机科学与探索, 2023, 17(10): 2389.(YANG B, SUN X H, DANG J Y, et al.Named entity recognition method of large language model for medical question answering system[J].Journal of frontiers of computer science and technology, 2023, 17(10): 2389.)
[27] SUN X, LI X, LI J, ET A L.Text classification via large language models[J].arXiv preprint arXiv: 2305.08377, 2023.
[28] ONG J, KEDIA N, HARIHAR S, et al.Applying large language model artificial intelligence for retina international classification of diseases (ICD) coding[J].Journal of medical artificial intelligence, 2023, 6: 21.
[29] WU L I, LI G.Zero-shot construction of Chinese medical knowledge graph with ChatGPT[C]//2023 IEEE international conference on medical artificial intelligence (MedAI).Beijing: IEEE, 2023: 278-283.
[30] ZHU Y, WANG X, CHEN J, et al.LLMs for knowledge graph construction and reasoning: Recent capabilities and future opportunities[J].arXiv preprint arXiv: 2305.13168, 2023.
[31] ZHU W, WANG X, CHEN M, et al.Overview of the PromptCBLUE shared tTask in CHIP2023[J].arXiv preprint arXiv: 2312.17522, 2023.
[32] LI B, FANG G, YANG Y, et al.Evaluating ChatGPT’s information extraction capabilities: an assessment of performance, explainability, calibration, and faithfulness[J].arXiv preprint arXiv: 2304.11633, 2023.
[33] WEI X, CUI X, CHENG N, et al.Zero-shot information extraction via chatting with ChatGPT[J].arXiv preprint arXiv: 2302.10205, 2023.
[34] MIALON, G.DESSÌ R, LOMELI M, et al.Augmented language models: A survey[J].arXiv preprint arXiv: 2302.07842, 2023.
[35] DUNN A, DAGDELEN J, WALKER N, et al.Structured information extraction from complex scientific text with fine-tuned large language models[J].arXiv preprint arXiv: 2212.05238, 2022.
[36] ZHENG C, GUO C Y.A pattern-first pipeline approach for entity and relation extraction[J].Neurocomputing, 2022(494): 182-191.
[37] XUE L L, ZHANG D, DONG Y X, et al.AutoRE: Document-level relation extraction with large language models[J].arXiv preprint arXiv: 2403.14888, 2024.
[38] 孙海霞, 郝洁, 郭臻, 等.基于知识元的细粒度医学量表文档知识表示框架构建[J].数字图书馆论坛, 2023, 19(12): 86-98.(SUN H X, HAO J, GUO Z, et al.Construction of a fine-grained knowledge element-based framework for knowledge representation in medical scale documents[J].Digital library forum, 2023, 19(12): 86-98.)
[39] 戴晓阳.常用心理评估量表手册[M].修订版.北京: 人民军医出版社, 2015: 150-154.(DAI X Y.Handbook of commonly used psychological assessment scales[M].Revised ed.Beijing: People’s Military Medical Publishing House, 2015: 150-154.)
[40] HUANG J W, YANG D H M, RONG R C, et al.A critical assessment of using ChatGPT for extracting structured data from clinical notes[J].NPJ digital medicine, 2024(7): 106.
[41] LANDIS J R, KOCH G G.The measurement of observer agreement for categorical data[J].Biometrics, 33(1): 159-174.
[42] NG A, FULFORD L.ChatGPT prompt engineering for developers[EB/OL].[2024-08-03].https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/.
[43] YUAN W, NEUBIG G, LIU P.Bartscore: Evaluating generated text as text generation[C]//Proceedings of the 34th international conference on neural information processing systems.New York: Curran Associates, 2021.
[44] CHEN J, LIU L, RUAN S, et al.Are different versions of ChatGPT’s ability comparable to the clinical diagnosis presented in case reports? a descriptive study[J].Journal of multidisciplinary healthcare, 2023, 16: 3825-3831.
Outlines

/

〈 〉