Scholarlink

<p>As structured data grows fast in dispersed contexts, storage economy, query latency, and failure tolerance are under pressure. Traditional Hadoop setups cannot handle schema evolution, access skew, and analytical workloads due to consistent replication and fixed file layouts. This work provides a constraint-aware multi-stage architecture for optimizing structured data storage using schema characteristics, metadata dependencies, and historical query behavior. After schema-profiling identifies attribute drift, redundancy, and cold access patterns, a hybrid reduction pipeline balances lossless and bounded lossy transformations for accuracy. To increase layout performance, a bilevel optimizer uses cost models from actual workload execution to evaluate row-group sizing, partitioning, and heterogeneous replication. A multi-objective evolutionary search chooses storage designs with good latency, footprint, and recovery stability. Real-world datasets and regulated query workloads assess the final architecture for analytical accuracy and system robustness under simulated node failures. A multi-terabyte transportation and aviation dataset showed significant storage usage reduction, consistent median and tail query latencies improvements, and faster fault recovery. To decrease metadata costs and scale business NameNodes, the design consolidates tiny file structures. A storage model with structural awareness, limited reduction, and evolutionary tuning improves performance and reliability for large-scale analytical systems. The method stresses end-to-end validation and integrated constraint handling for next-generation distributed platform data storage. </p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Design of an iterative method with constraint-aware structured data storage optimization for Hadoop using an efficient scalable multi-stage analytical framework process

  • 期刊论文

  • 发表日期：

    2025年12月27日

查看全文
  • Samita Patil,
  • Savita Sangam

摘要

As structured data grows fast in dispersed contexts, storage economy, query latency, and failure tolerance are under pressure. Traditional Hadoop setups cannot handle schema evolution, access skew, and analytical workloads due to consistent replication and fixed file layouts. This work provides a constraint-aware multi-stage architecture for optimizing structured data storage using schema characteristics, metadata dependencies, and historical query behavior. After schema-profiling identifies attribute drift, redundancy, and cold access patterns, a hybrid reduction pipeline balances lossless and bounded lossy transformations for accuracy. To increase layout performance, a bilevel optimizer uses cost models from actual workload execution to evaluate row-group sizing, partitioning, and heterogeneous replication. A multi-objective evolutionary search chooses storage designs with good latency, footprint, and recovery stability. Real-world datasets and regulated query workloads assess the final architecture for analytical accuracy and system robustness under simulated node failures. A multi-terabyte transportation and aviation dataset showed significant storage usage reduction, consistent median and tail query latencies improvements, and faster fault recovery. To decrease metadata costs and scale business NameNodes, the design consolidates tiny file structures. A storage model with structural awareness, limited reduction, and evolutionary tuning improves performance and reliability for large-scale analytical systems. The method stresses end-to-end validation and integrated constraint handling for next-generation distributed platform data storage.