The railway industry has accumulated an enormous amount of heterogeneous data from various sources over years of development. Effectively storing, managing, and fully leveraging these vast and complex data resources has become a critical issue. As a storage architecture capable of efficiently handling diverse types of data, data lakes have been widely implemented across multiple sectors, such as finance, healthcare, and manufacturing. However, with the continued growth in the volume of data stored within data lakes and the increase in data access frequency, traditional static storage strategies have begun to show limitations, specifically in terms of low access efficiency and high storage costs. To address this, this paper proposes a dynamic data resource status awareness technology for data lakes. By monitoring data access logs in real time and analyzing trends in data access frequency and usage scenarios, this technology intelligently identifies the “hot” or “cold” status of data. High-frequency accessed data is migrated to high-performance storage media, while low-frequency data is archived in more cost-effective storage devices. Through dynamically adjusting the storage locations of data, this approach optimizes the use of storage resources, enhances overall storage efficiency, and reduces operational costs.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Research on Intelligent Sensing Technology for Data Resource Status in Railway Big Data Lakes

  • Siqi Sun,
  • Kai Zhang,
  • Dan Zou,
  • Guohua Li,
  • Yanjun Liu

摘要

The railway industry has accumulated an enormous amount of heterogeneous data from various sources over years of development. Effectively storing, managing, and fully leveraging these vast and complex data resources has become a critical issue. As a storage architecture capable of efficiently handling diverse types of data, data lakes have been widely implemented across multiple sectors, such as finance, healthcare, and manufacturing. However, with the continued growth in the volume of data stored within data lakes and the increase in data access frequency, traditional static storage strategies have begun to show limitations, specifically in terms of low access efficiency and high storage costs. To address this, this paper proposes a dynamic data resource status awareness technology for data lakes. By monitoring data access logs in real time and analyzing trends in data access frequency and usage scenarios, this technology intelligently identifies the “hot” or “cold” status of data. High-frequency accessed data is migrated to high-performance storage media, while low-frequency data is archived in more cost-effective storage devices. Through dynamically adjusting the storage locations of data, this approach optimizes the use of storage resources, enhances overall storage efficiency, and reduces operational costs.