Given coal energy’s continued dominance in China’s industry, precise coal powder particle segmentation technology is critical for improving resource sorting efficiency. Mainstream semantic segmentation models, however, are constrained by rigid geometric assumptions. This makes them ill-suited for the complex, irregular morphologies of coal powder, especially encountering representation bottlenecks with industrial challenges like edge fractures and particle adhesion. This paper proposes the innovative QA-TransUNet framework, redefining feature modeling via a geometric adaptive attention mechanism called ‘Quadrangle Attention’. It deeply integrates deformable vision theory with the Transformer architecture, replacing traditional rectangular sampling with dynamically deformable windows. These windows intelligently adapt their shape based on target contours, driven by a lightweight parameter network enabling adaptive rotation, scaling, and perspective adjustments. A biomimetic feature remapping strategy further enhances the model’s ability to preserve the topological structure of quadrilateral-like particles. Systematic experiments demonstrate QA-TransUNet achieves a paradigm-shifting improvement in segmentation accuracy for complex industrial environments, paving new technical pathways for geometry-sensitive tasks in mineral sorting, biomedical imaging, and related fields.