The building sector accounts for almost 40% of global energy consumption. However, buildings’ Heating, Ventilation, and Air Conditioning (HVAC) systems are susceptible to various faults and defects, causing significant declines in buildings’ energy efficiency. This implies a critical demand for suitable fault detection and diagnosis methods. At the same time, Large Language Models (LLMs) evolved rapidly over the last few years and, especially since the release of ChatGPT in 2022, gained widespread attention. LLMs interpret and process natural-language content as sequence-like data. We utilize LLMs proficiency in dealing with sequential input to handle time series data of buildings’ HVAC systems and develop a novel fault detection method, harnessing LLMs to detect faults in common HVAC systems, thereby helping to mitigate energy wastage in buildings. We use publicly available time series datasets from a collection of European buildings’ most common HVAC systems, serialize them, and pass them to a pre-trained LLM (DistilBERT). By fine-tuning the model with a large number of labeled input data, we enable classification into either binary cases (faulty/fault-free) or multiple fault classes. The performance is assessed using a 5-fold time series cross-validation, yielding an F1-score between 82–99% for the binary fault classification and a macro-averaged F1-score of up to 99% for the multi-class classification tasks. The main advantage of using LLMs for fault detection is that in contrast to conventional fault detection methods, LLMs can naturally deal with noisy input data. This can reduce the required preprocessing steps such as removing randomly missing values, encoding categorical features, and normalization.