The classification of aerial scene images often depends on resource-intensive deep learning models, making its deployment on embedded or edge devices challenging. For time-critical applications like emergency response, a lightweight model with low computational cost and fast inference time is essential. In this work, we optimize the EmergencyNet model by incorporating channel pruning strategy to reduce model complexity while maintaining performance. Furthermore, the use of orthogonal initialization enhances the model’s robustness during training. Experimental results on two benchmark aerial scene datasets shows that the proposed modifications not only reduces the model’s size, but also improve the classification accuracy as compared to several state-of-the-art lightweight modeling techniques, making it more suitable for real-time deployment in resource-constrained environments. Experiments on two benchmark datasets show that with only a cost of a 5–8% performance drop, our proposed approach is able to achieve a model size reduction of more than 87% using the 60% ratio of EmergencyNet channel pruning.