Earth retaining structures (ERS) play a crucial role in the infrastructure of railway networks. It supports and protects railway networks from soil and rock erosion. However, different types of defects can form on ERS that can cause slope failure, soil erosion, and degradation of its structural integrity. Timely repair of these defects is essential. However, before detecting defects, it is important to classify ERS, as the occurrence and nature of defects often depend on the specific type of ERS. This study proposes the use of artificial intelligence (AI), specifically improved ConvNeXt V2, for automatically classifying three common types of railway ERS captured with GPS metadata from Transport for New South Wales (TfNSW) railway networks as a proof of concept. Classifying ERS is a necessary step prior to defect detection because defects of ERS are dependent on its type. The study also extracts the GPS coordinates of the classified ERS from the captured data to enable the inspector to locate and conduct further inspection of that ERS.