With the rapid development of autonomous driving, high accuracy and low latency are required for traffic sign detection, especially for small objects. However, achieving a balance between accuracy and speed is challenging due to excessive backbone computation, inadequate feature representation, and the design of separate tasks. FTC-YOLO is proposed to address these problems based on feature enhancement and task calibration. First, the network is enhanced by removing large detection layers that introduce unnecessary computation and by adding new layers for small objects so that detection performance is improved while efficiency is maintained. Then, an innovative StarNet backbone is utilized, employing star operations for rapid feature extraction. A Semantic-Detail Feature Enhancement Module (SFEM) is also designed, utilizing frequency domain transformations and global information to capture context from multiple directions, thereby enhancing the representation of small objects. Finally, a Joint Representation Calibration Detection Head (JRCDH) is proposed, which utilizes shared convolutions to enhance feature consistency across different representations. With deformable convolution (DCNv2) and channel attention, the alignment between classification and localization is improved. The bounding box loss function is further adjusted by combining Wise-IoU with the normalized Gaussian Wasserstein distance, making localization more stable and better at handling object shapes. On the TT100K dataset, FTC-YOLO achieves a mAP of 83.4%, representing a 9.8% improvement over YOLOv8n while reducing the number of parameters by 59%. The detection accuracy for small objects is also improved to 48.8%. On the GTSDB dataset, the overall mAP@0.5:0.95 reaches 81.6%, significantly outperforming YOLOv8s (81.0%).