4D point cloud video semantic segmentation is essential for dynamic 3D scene understanding but remains challenging due to data sparsity and lack of texture in point clouds. Existing methods, primarily based on geometric structures, are often limited in spatiotemporal modeling. To overcome these limitations, we propose a novel dual-branch cross-modal fusion network that leverages complementary information from RGB videos. Our framework features a dedicated RGB branch which separately extracts texture features and temporal gradient features. These are then fused by an internal cross-attention module to enhance the representation of appearance and motion. Subsequently, a cross-modal Transformer aligns and integrates these enriched RGB features with the spatiotemporal features from the point cloud branch—the latter modeled via self-attention mechanisms. Experiments on the Synthia 4D and HOI4D datasets demonstrate the superior performance of our approach in 4D semantic segmentation, which is supported by comprehensive ablation studies validating the contribution of each module.