Fine-grained toxic language detection is of growing significance for maintaining healthy media platforms and social environments. With diverse transitions and metaphors, informal and unstructured social media texts hinder the discernment of toxicity and offensiveness towards specific identities. Additionally, due to this misunderstanding, existing methods lack exploration of relationships between features of different granularities, such as toxic type and attack target. This hinders further improvement of detection performance. To address these issues, we propose a Fine-grained Toxicity Analysis Framework (FTAF) to simplify and normalize text expression and promote the detection of fine-grained toxic language. First, we construct a Social Media Text Normalizer (SMTN) to simplify texts into analyzable structures, aiding in the capture of hate speech features. Subsequently, we develop a COT style Relationship Chain Fine-Tuning (RCFT) method that detect toxicity by learning the structured relationships between toxicity features. Extensive experiments demonstrate that our proposed FTAF method enhances the performance of fine-grained toxic language detection.