Design of AI Agent Models Driven by Multimodal Data

Han Xuewen; Che Shangkun; Yang Mengqing; Wang Neng

doi:10.13266/j.issn.0252-3116.2024.24.003

Library and Information Service >

2024 , Vol. 68 >Issue 24: 27 - 37

DOI: https://doi.org/10.13266/j.issn.0252-3116.2024.24.003

SPECIAL TOPIC: Research on Security Situation Awareness Driven by Multimodal Data

Design of AI Agent Models Driven by Multimodal Data

Han Xuewen ,
Che Shangkun ,
Yang Mengqing ,
Wang Neng

Expand

1 School of Economics and Management, Tsinghua University, Beijing 100084;
2 School of Journalism and Communication, Nanjing Normal University, Nanjing 210023;
3 Department of Electrical and Computer Engineering, University of California, Los Angeles 90095

Han Xuewen, doctoral candidate;Che Shangkun, doctoral candidate;Yang Mengqing, lecturer, PhD, correspondence author, E-mail:mqyang@nnu.edu.cn;Wang Neng, postgraduate student.

Received date: 2024-06-21

Revised date: 2024-09-28

Online published: 2024-12-23

Supported by

This work is supported by the general project of the National Social Science Fund of China titled “Research on Multimodal Network Data Security Situation Awareness and Risk Collaborative Governance Mechanism” (Grant No. 23BTQ076).

Fold

Abstract

[Purpose/Significance] This study develops an AI agent model for multimodal data processing across varied tasks. By exploring the inherent value of multimodal data, the model enhances the support for decision-making processes. [Method/Process] This study designed a hybrid AI agent model that integrated both horizontal and vertical structures. Using the financial annual reports from thirty Dow Jones Industrial Average companies as a case study, this model facilitated intelligent investment research and analysis for the year 2023. The research compared the performance of different organizational structures of financial agents in executing three subtasks: analysis of company fundamentals, market sentiment, and risk and opportunity assessment. Decisions were made based on these analyses, leveraging large language models capable of processing multimodal data. [Result/Conclusion] The vertical agent structure excels in tasks requiring efficient, directional strategic analysis with clear hierarchical decision-making. Conversely, the horizontal structure is more effective in scenarios demanding collaboration, flexibility, and broad expertise. The hybrid model outperforms single-structure agents in terms of accuracy in the final investment decision, demonstrating its efficacy in optimizing task-specific agent configurations.

Key words： multi-modal data; AI agents; horizontal agent structure; vertical agent structure; financial decision

Cite this article

Han Xuewen , Che Shangkun , Yang Mengqing , Wang Neng . Design of AI Agent Models Driven by Multimodal Data[J]. Library and Information Service, 2024 , 68(24) : 27 -37 . DOI: 10.13266/j.issn.0252-3116.2024.24.003

References

[1] 黄永文, 孙坦, 赵瑞雪, 等. 大数据与人工智能背景下新型知识服务研究与实践[J]. 图书情报工作, 2022, 66(19): 36-46. (HUANG Y W, SUN T, ZHAO R X, et al. Research and practice of new knowledge service in the context of big data and artificial intelligence[J]. Library and information service, 2022, 66(19): 36-46.)
[2] 祁彬斌, 张文梅, 朱学芳, 等. 面向馆藏数字资源的多模交互模型与影响因素研究[J]. 图书情报工作, 2023, 67(5): 90-98. (QI B B, ZHANG W F, ZHU X F, et al. Research on multi-mode interaction model and influencing factors for digital collections[J]. Library and information service, 2023, 67(5): 90-98.)
[3] 张利锋. 基于神经网络的医疗数据处理关键技术研究[D]. 北京: 北京邮电大学, 2024. (ZHANG L F. Research on key technologies of medical data processing based on neural network[D]. Beijing: Beijing University of Posts and Telecommunications, 2024.)
[4] MAITINIYAZI M, VASIT S, PAHEDING S, et al. Soybean yield prediction from UAV using multimodal data fusion and deep learning[J]. Remote sensing of environment, 2020, 237: 111599.
[5] ZHANG Y D, DONG Z, WANG S H, et al. Advances in multimodal data fusion in neuroimaging: overview, challenges, and novel orientation[J]. Information fusion, 2020, 12(64): 149-187.
[6] 熊回香, 杨滋荣, 蒋武轩.跨媒体知识图谱构建中多模态数据语义相关性研究[J]. 情报理论与实践, 2019, 42(2): 13-18, 24. (XIONG H X, YANG Z R, JIANG W X. Semantic correlation of multimodal data in the construction of cross-media knowledge graph[J]. Information studies: theory & application, 2019, 42(2): 13-18, 24.)
[7] 李纲, 王施运, 毛进, 等.面向态势感知的国家安全事件图谱构建研究[J]. 情报学报, 2021, 40(11): 1164-1175. (LI G, WANG S Y, MAO J, et al. Construction of national security event map and its application for situation awareness[J]. Journal of the China Society for Scientific and Technical Information, 2021, 40(11): 1164-1175.)
[8] 马超, 李纲, 陈思菁, 等. 基于多模态数据语义融合的旅游在线评论有用性识别研究[J]. 情报学报, 2020, 39(2): 199-207. (MA C, LI G, CHEN S J, et al. Research on usefulness recognition of tourism online reviews based on multimodal data semantic fusion[J]. Journal of the China Society for Scientific and Technical Information, 2020, 39(2): 199-207.)
[9] SHI L, YANG X, LI J, et al. Scenario construction and deduction for railway emergency response decision-making based on network models[J]. Information sciences, 2022, 588: 331-349.
[10] FARIDA M, HAZRAT A, NADY E H, et al. Artificial intelligence-based methods for fusion of electronic health records and imaging data[J]. Scientific reports, 2022, 12(1): 17981.
[11] 吴雅威, 龚蛟腾, 朱洪涛.面向科技智库决策咨询研究需求的多元化资源融合与服务路径[J]. 图书情报工作, 2023, 67(5): 78-89. (WU Y W, GONG J T, ZHU H T. Diversified resource integration and service path for decision-making consultation research needs of scientific and technological think tanks[J]. Library and information service, 2023, 67(5): 78-89.)
[12] 王松, 焦海燕, 刘新民.融合外部知识图谱的多模态知识共创价值识别研究[J]. 情报理论与实践, 2024, 47(11): 139-149. (WANG S, JIAO H Y, LIU X M. Research on multi-modal knowledge co-creation value recognition by integrating external knowledge graph[J]. Information studies: theory & application, 2024, 47(11): 139-149.)
[13] 韩普, 陈文祺, 顾亮, 等.融合多模态数据的中文医学实体识别研究[J]. 情报理论与实践, 2024, 47(9): 174-182. (HAN P, CHEN W Q, GU L et al. Research on Chinese medical named entity recognition with fusion of multimodal data[J]. Information studies: theory & application, 2024, 47(9): 174-182.)
[14] 金光, 包阳.基于大模型的智能系统试验数据分析技术初探[J/OL]. 系统工程与电子技术:1-13[2024-05-20]. http://kns.cnki.net/kcms/detail/11.2422.TN.20240517.1849.015.html. (JIN G, BAO Y. Research on test data analysis for intelligent system using LLMs[J/OL]. Systems engineering and electronics:1-13[2024-05-20]. http://kns.cnki.net/kcms/detail/11.2422.TN.20240517.1849.015.html.)
[15] MASTERMAN T, BESEN S, SAWTELL M, et al. The Landscape of emerging AI agent architectures for reasoning, planning, and tool calling: a survey[J/OL]. arXiv preprint, 2024, arXiv: 2404.11584.
[16] 张连福, 谭作文.一种面向多模态医疗数据的联邦学习隐私保护方法[J]. 计算机科学, 2023, 50(S2): 933-940. (ZHANG L F, TAN Z W. Federated learning privacy-preserving approach for multimodal medical data[J]. Computer science, 2023, 50(S2): 933-940.)
[17] AMAL S, SAFARNEJAD L, OMIYE J A, et al. Use of multi-modal data and machine learning to improve cardiovascular disease care[J]. Frontiers in cardiovascular medicine, 2022, 9:840262.
[18] 陈孟, 干可, 李凯, 等.基于实时多模态时空数据的时空图卷积网络精准鲁棒交通流预测模型[J]. 公路交通科技, 2021, 38(8): 134-139, 158. (CHEN M, GAN K, LI K, et al. A Spatial-temporal graph convolutional network model for accurate and robust traffic flow prediction based on real-time multimodal spatial-temporal data[J]. Journal of highway and transportation research and development, 2021, 38(8): 134-139, 158.)
[19] SMARZARO R, Davis JR C A, QUINTANILHA J A. Creation of a multimodal urban transportation network through spatial data integration from authoritative and crowdsourced data[J]. ISPRS International journal of geo-information, 2021, 10(7): 470.
[20] EISELE W L, RILETT L R, BROWN MHOON, K, et al. Using intelligent transportation systems travel-time data for multimodal analyses and system monitoring[J]. Transportation research record, 2001, 1768(1): 148-156.
[21] YANG H, ZHANG B, WANG N, et al. FinRobot: An open-source AI agent platform for financial applications using large language models[J]. arXiv preprint, 2024, arXiv: 2405.14767.
[22] 李佳轩, 储节旺, 杜秀秀.关联、黑箱与赋能: AIGC驱动智慧图书馆的转型路径[J]. 图书情报工作, 2023, 67(23): 18-27. (LI J X, CHU J W, DU X X. Connection, black box and empowerment: the path of AIGC to drive the transformation of smart libraries[J]. Library and information service, 2023, 67(23): 18-27.)
[23] HIRIYANNAIAH S, SRINIVAS A M D, SHETTY G K, et al. A computationally intelligent agent for detecting fake news using generative adversarial networks[M]. Hybrid computational intelligence. San Diego: Academic Press, 2020: 69-96.
[24] YANG H. FinRobot [EB/OL]. [2024-09-17]. https://github.com/AI4Finance-Foundation/FinRobot.
[25] HUGGINGFACE. Pretrained models [EB/OL]. [2024-09-17]. https://www.sbert.net/docs/pretrained_models.html.
[26] 汪弘, 罗党论, 林东杰.行业分析师的研究报告对投资决策有用吗?——来自中国A股上市公司的经验证据[J]. 证券市场导报, 2013(7): 36-43. (WANG H, LUO D L, LIN D J. Are industry analysts’ research reports useful for investment decisions? empirical evidence from Chinese A-share listed companies[J]. Securities market herald, 2013(7): 36-43.)
[27] ABARBANELL J S, BUSHEE B J. Fundamental analysis, future earnings, and stock prices[J]. Journal of accounting research, 1997, 35(1): 1-24.
[28] BOLLEN J, MAO H, ZENG X. Twitter mood predicts the stock market [J]. Journal of computational science, 2011, 2(1): 1-8.
[29] HULL J C. The evaluation of risk in business investment[M]. Berkeley: Elsevier, 2014.
[30] HOUCINE A. The effect of financial reporting quality on corporate investment efficiency: evidence from the Tunisian stock market[J]. Research in international business and finance. 2017, 42: 321-337.
[31] ERTIMUR Y, SUNDER J, SUNDER S V. Measure for measure: the relation between forecast accuracy and recommendation profitability of analysts[J]. Journal of accounting research, 2010, 45(3): 567-606.
[32] ZHANG H, GONG J, WU W. Artificial intelligence fortext generation: an intellectual property perspective[C]// International conference on AI-generated content. Singapore: Springer Nature Singapore. 2023: 266-279.
[33] ZENG J, HUANG R, MALIK W, et al. Large language models for social networks: applications, challenges, and solutions[J]. arXiv preprint, 2024, arXiv: 2401.02575.

Options

Outlines

模态框（Modal）标题

Abstract

Cite this article

References