A spatial big data architecture for vehicle data analytics: a focus on spatial data storage and query optimization
摘要
Effective management of vehicle mobility data is critical for advancing smart urban mobility and transportation analytics. Beyond data collection, the field requires integrated, end-to-end systems to handle large, heterogeneous datasets generated by vehicle sensors. Challenges include managing extensive data volumes, addressing data heterogeneity, ensuring robust storage and scalable processing, and complying with strict data protection regulations for sensitive location-based information. Additionally, managing the computational complexity of spatial data and efficiently handling metadata is vital for meaningful analysis. To address these challenges, this paper introduces a comprehensive Big Data architecture designed for large-scale, heterogeneous vehicle trajectory data processing. It evaluates architectural and storage design choices, focusing on high-dimensional spatial query optimization. Techniques such as space-filling curves (Z-order and Hilbert) and bucketing methods (H3 and S2 indexing) significantly improved query performance, reducing range query times by over 50% and join query times by up to 80%. Further enhancements were achieved with Cost-Based Optimization, Adaptive Query Execution, and Dynamic File Pruning. This paper outlines the key elements in building an end-to-end solution for spatial data analytics within the domain of vehicle mobility data analytics. It highlights the critical aspects and challenges of this field, offering insights into the development of an effective data management system tailored to the unique demands of smart urban mobility.