Objects with close distance in optical remote sensing images often exhibit correlation, such as cars, and ships. However, existing methods generally tackle the perception of objects in optical remote sensing independently. To this end, we propose a Group Object Perception framework (GOP), which utilizes the correlation between objects within the group to improve the perception precision. As a perception framework, GOP is comprehensive including category, bounding box, and mask information. Firstly, we define the group object mainly considering spatial proximity and design an automated labeling scheme to produce labels for GOP. Then we propose a baseline model to detect and segment group objects and sub-objects simultaneously. Furthermore, to utilize the correlation between objects in the group, we propose position supervisor predicting offsets from individual objects to the corresponding group object, which can enhance the detection performance. Additionally, we introduce group category learner to cluster feature similarity from the same class within the group object, further improving the segmentation results. In experiments, GOP obtains 78.2 mAP and 75.9 mAP in rotated object detection and object segmentation on DOTA-v1.0 dataset, respectively. We will release our source codes, pre-trained models, and online demos to facilitate further studies.