Accelerating I/O in Scientific Workflows with the Impact of Apache Ignite’s In-Memory File System
摘要
Efficient storage and retrieval of large-scale scientific data remain critical challenges in HPC environments, especially as heterogeneous architecture and evolving system complexities continue to increase demands on application developers. This work-in-progress paper investigates the potential of in-memory computing file systems, specifically Ignite’s Distributed File System (IDFS), for mitigating I/O bottlenecks in scientific workflows. We evaluate IDFS performance by single-node deployment at the NERSC Perlmutter supercomputer for latency in data access, throughput, and resource utilization. Our preliminary results prove the feasibility of using IDFS to improve I/O efficiency in large-scale scientific applications. This work adds to the current efforts in scalable data management and performance optimization, hence setting the base for future multi-node evaluations and integration with advanced parallel and distributed computing methodologies.