With the intelligent and connected development of the automotive industry, the types of data generated by vehicles are increasingly complex, including structured and unstructured data. The traditional centralized data processing architecture is difficult to meet the real-time analysis requirements of massive multi-source heterogeneous data. This paper presents an automotive product insight system based on the kafka + spark distributed architecture. First, this paper forms a multi-source heterogeneous data system by collecting multi-source heterogeneous data. Secondly, a hierarchical design approach is adopted to construct a distributed processing framework that includes data acquisition layer, distributed storage layer, data cleaning layer, memory buffer layer, etc. Then, the system uses Kafka for real-time data stream processing and Spark for batch processing analysis; Finally, the insights can be presented through a visual interface, enabling functions such as vehicle performance monitoring and market trend prediction. Experiments show that the system can effectively improve data processing efficiency and analysis accuracy, providing decision support for automakers and mobility service providers.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

An Automotive Product Insight System Based on the Kafka+Spark Distributed Architecture

  • Huiying Hu,
  • Yishu Zhao,
  • Fan Zhang

摘要

With the intelligent and connected development of the automotive industry, the types of data generated by vehicles are increasingly complex, including structured and unstructured data. The traditional centralized data processing architecture is difficult to meet the real-time analysis requirements of massive multi-source heterogeneous data. This paper presents an automotive product insight system based on the kafka + spark distributed architecture. First, this paper forms a multi-source heterogeneous data system by collecting multi-source heterogeneous data. Secondly, a hierarchical design approach is adopted to construct a distributed processing framework that includes data acquisition layer, distributed storage layer, data cleaning layer, memory buffer layer, etc. Then, the system uses Kafka for real-time data stream processing and Spark for batch processing analysis; Finally, the insights can be presented through a visual interface, enabling functions such as vehicle performance monitoring and market trend prediction. Experiments show that the system can effectively improve data processing efficiency and analysis accuracy, providing decision support for automakers and mobility service providers.