With the increasing complexity of the network environment, multi-source heterogeneous data has brought more challenges to the traditional intrusion detection system. In this paper, we propose and implement a big data intrusion detection model (MSF-IDM) based on multi-source data fusion to solve the problems of high data dimensionality, semantic inconsistency, and strict real-time requirements. The model adopts a mid-level fusion strategy combined with a cross-attention mechanism to achieve semantic alignment and deep fusion of network traffic, system logs and user behavior data. At the system level, a distributed stream batch processing integration platform based on Kafka, Spark Streaming and PyTorch is constructed to ensure high throughput and low latency. Experimental evaluations on multiple datasets, including NSL-KDD, UNSW-NB15 and CICIDS2017, show that MSF-IDM outperforms traditional methods in terms of detection accuracy, F1 score, real-time response and resource consumption. These results validate the effectiveness and practicality of the model and provide a systematic solution for large-scale intrusion detection in multi-source fusion environments.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Design and Evaluation of Big Data Intrusion Detection Model Based on Multi-Source Data Fusion

  • Kunjian Tang

摘要

With the increasing complexity of the network environment, multi-source heterogeneous data has brought more challenges to the traditional intrusion detection system. In this paper, we propose and implement a big data intrusion detection model (MSF-IDM) based on multi-source data fusion to solve the problems of high data dimensionality, semantic inconsistency, and strict real-time requirements. The model adopts a mid-level fusion strategy combined with a cross-attention mechanism to achieve semantic alignment and deep fusion of network traffic, system logs and user behavior data. At the system level, a distributed stream batch processing integration platform based on Kafka, Spark Streaming and PyTorch is constructed to ensure high throughput and low latency. Experimental evaluations on multiple datasets, including NSL-KDD, UNSW-NB15 and CICIDS2017, show that MSF-IDM outperforms traditional methods in terms of detection accuracy, F1 score, real-time response and resource consumption. These results validate the effectiveness and practicality of the model and provide a systematic solution for large-scale intrusion detection in multi-source fusion environments.