Scholarlink

Agent-based modeling demands many interdisciplinary steps for development, which requires people with different skills, such as simulationists and non-simulationists. The different skill sets of these role players, especially non-simulations (thematician, theoretician) and simulations (programmers), can cause inconsistent documents and missing information. Even though the challenge of dealing with text narratives can be observed in most steps of ABM development (the comprehensive analysis of narratives, conceptual model generation, model documentation, and etc.), different methods with a specific focus on each of these steps have been suggested, such as using standards like ODD for better document structuring, UMLs for illustration, and natural language modeling for direct text-to-code conversion. The objective of this work is to look at the opportunities rendered by Question-Answer models to support the extraction of model information from conceptual narratives (conceptual model), e.g., to render an understanding of the simulation model described in formal or informal terms, serving as a starting point to draw on such techniques to foster collaborative modeling based on diverse data sources, which in addition can be used in an executable model generation. Emphasizing the initial objective, in this paper, we propose a benchmark and roadmap for extracting model information from conceptual models and presenting it in JSON format (both human and computer-readable format), which can be useful to reduce the communication gaps between simulationists and non-simulationists. This benchmark includes the specification of general ABM features to be extracted, as well as the selection of simulation model samples, showcasing variability in feature set and formality. Our findings indicate that among a set of 13 open-source and 3 commercial models, to date only commercial models provide tangible achievements. Specifically, OpenAI’s recently released ChatGPT-4O model offers the strongest performance across our sample (average precision rate: 84%) compared to other models such as ChatGPT-4.5-turbo (57%) and ChatGPT-3.5 (40%). We discuss opportunities and implications, both for the use of QA models for information extraction purposes and future developments, as well as the need for pre-training and fine-tuning open-source models to be meaningfully engaged in this effort.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Toward Automating Agent-Based Model Generation: A Benchmark for Model Extraction Using Question-Answering Techniques

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Siamak Khatami,
  • Christopher Frantz

摘要

Agent-based modeling demands many interdisciplinary steps for development, which requires people with different skills, such as simulationists and non-simulationists. The different skill sets of these role players, especially non-simulations (thematician, theoretician) and simulations (programmers), can cause inconsistent documents and missing information. Even though the challenge of dealing with text narratives can be observed in most steps of ABM development (the comprehensive analysis of narratives, conceptual model generation, model documentation, and etc.), different methods with a specific focus on each of these steps have been suggested, such as using standards like ODD for better document structuring, UMLs for illustration, and natural language modeling for direct text-to-code conversion. The objective of this work is to look at the opportunities rendered by Question-Answer models to support the extraction of model information from conceptual narratives (conceptual model), e.g., to render an understanding of the simulation model described in formal or informal terms, serving as a starting point to draw on such techniques to foster collaborative modeling based on diverse data sources, which in addition can be used in an executable model generation. Emphasizing the initial objective, in this paper, we propose a benchmark and roadmap for extracting model information from conceptual models and presenting it in JSON format (both human and computer-readable format), which can be useful to reduce the communication gaps between simulationists and non-simulationists. This benchmark includes the specification of general ABM features to be extracted, as well as the selection of simulation model samples, showcasing variability in feature set and formality. Our findings indicate that among a set of 13 open-source and 3 commercial models, to date only commercial models provide tangible achievements. Specifically, OpenAI’s recently released ChatGPT-4O model offers the strongest performance across our sample (average precision rate: 84%) compared to other models such as ChatGPT-4.5-turbo (57%) and ChatGPT-3.5 (40%). We discuss opportunities and implications, both for the use of QA models for information extraction purposes and future developments, as well as the need for pre-training and fine-tuning open-source models to be meaningfully engaged in this effort.