Semantic segmentation is a crucial task for autonomous driving and intelligent transportation systems, requiring high accuracy and low latency. Traditional architectures based solely on convolutional neural networks (CNNs) or Transformers have notable limitations: CNNs lack global perception capabilities, while Transformers incur high computational overhead and are challenging to deploy on edge devices. To address these issues, this paper proposes a lightweight CNN-Transformer hybrid model and designs a customized accelerator for FPGA to achieve real-time semantic segmentation. The model uses a deformable attention module to capture global context, combines deep separable convolution to improve the efficiency of local feature extraction, and introduces a skip connection enhancement module to restore edge details, taking into account both segmentation accuracy and computational overhead. To adapt to resource-constrained platforms, the accelerator optimizes parallel computing, memory access, and data flow scheduling through 8 bit quantization, operator decomposition, and 3D systolic array design. Experimental results show that the system achieves 72.8% mIoU on the Cityscapes dataset and an inference speed of 67 FPS, which is better than the mainstream CPU/GPU platforms in terms of energy efficiency and latency. This study provides an efficient and feasible solution for the deployment of complex neural networks in edge devices with a wide application potential.