Steel is an integral part of the infrastructure. Nowadays, steel has become a popular material in infrastructure due to its reduced processing time and easy fitting. Rust damage is one of the major causes of the degradation of steel infrastructure, and effective monitoring of rust is crucial for evaluating the damage to steel structures. The purpose of this research is to develop a near real-time rust recognition method for steel structure health monitoring using transformers-based Convolutional Neural Networks, e.g., SegFormer. The proposed transformer-based CNN rust recognition method is a simple, efficient, and powerful solution for semantic segmentation, which utilizes transformers and lightweight multilayer perceptron (MLP) decoders to reduce processing time and enhance accuracy. Conventional manual rust inspection is time-consuming, sometimes dangerous, and prone to human mistakes. The transformer-based CNN method could tackle these problems and minimize the costly manpower requirement for inspection. Most deep learning models for semantic segmentation were developed solely based on convolutional architecture and required explicit positional encoding. Consequently, the resolution variance between the testing and training phases led to reduced performance. In contrast, SegFormer is a hybrid architecture that integrates components from both transformers and convolutional neural networks (CNNs) and does not require positional encoding. In this research, the proposed method adopts the “light PyTorch” backend to significantly reduce the required processing time and develop a near real-time rust recognition method. It is able to quickly process rust segmentation at the image pixel level, and provide fast rusting deterioration assessment of steel structures.