Arbitrary-oriented multi-scale objects and complex background information make object detection in remote sensing images highly challenging. To address this, we propose a novel two-stage oriented object anchor-free detection network (TOAF-Det). Specifically, we fine tune a supervised attention module to learn key features of oriented objects and reduce interference from complex backgrounds. To eliminate the dependency on predefined anchor shapes, we introduce a IoU prediction branch to the anchor-free oriented region proposal network to improve the model's adaptability to multi-scale objects. Finally, extensive experiments on the DOTA dataset demonstrate the effectiveness of our method.