This paper presents a novel and efficient multitask learning for scene classification and object detection from remote sensing images, which mainly contains the Non-Independent and Identically Distributed Fisher Vector Network (NIID-FVNet) for scene classification and Two-Stream CNN-Transformer Network (TSCTNet) for object detection. NIID-FVNet is designed to classify the input image into inshore or offshore scene. Then, TSCTNet is proposed for extracting more informative salient and edge features, in which a three-stream decoder consisting of a saliency stream, edge stream and feature fusion stream is introduced to enhance the detection result by taking full advantage of the extracted information from different modalities. Finally, the salient-aware module and edge-aware module are designed to generate more accurate saliency detection results with clear boundaries. Experimental results demonstrate that the presented scene classification and object detection networks outperforms other approaches.