Federated learning enhances data privacy but faces security risks like model theft, while traditional cryptographic methods often compromise efficiency or accuracy. Although Trusted Execution Environments (TEEs) offer hardware-level security, their limited memory and lack of hardware acceleration hinder deep neural network deployment. Current partitioning solutions alleviate memory constraints but introduce new vulnerabilities, highlighting the need for lightweight TEE optimization. We propose FedDualPrune, a lightweight federated learning framework that leverages a TEE-based dual-branch architecture. During local training, a frozen pre-trained model in the rich execution environment (REE) serves as a general feature extractor, while a unidirectional feature fusion mechanism securely integrates its outputs with a trainable counterpart inside the TEE. Furthermore, a channel-interaction-aware joint pruning strategy structurally compresses the TEE-hosted branch to satisfy secure memory constraints while preserving model performance. Extensive experiments show that under high pruning rates and Non-IID conditions, the accuracy loss is controlled within 5%, achieving synergistic optimization of privacy protection and model performance.