Given the surge in network attack behavior, detecting malicious traffic has become a pivotal cybersecurity task. Many existing methods for malicious traffic detection rely on machine learning and deep learning, but they exhibit certain shortcomings: A considerable number of these methods rely on statistical features, which may lose their relevance as networks evolve and lead to the loss of important information. Additionally, Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) face limitations in extracting features from network traffic, specifically, their inability to capture traffic interaction behavior information within a network flow. In this paper, we propose a Traffic Behavior Analysis model with Graph Neural Networks (TBA-GNN), which works directly with raw bytes and leverages the hierarchy structure of traffic (byte-packet-flow) to delve into valuable information. Firstly, we devise the PacketCNN to extract packet-level features from raw bytes. Subsequently, we construct a network flow as a Traffic Interaction Graph, containing both the traffic interaction behavior information and packet-level traffic information, and utilize the GNN to extract flow-level features. Finally, we perform a classification task to detect malicious traffic. We conduct extensive experiments on the ISCXIDS2012 and CICIDS2017 datasets, and the experimental results demonstrate that our model effectively identifies malicious traffic, outperforming baselines significantly.