Buildings consume significant energy, and enhancing their energy efficiency can have a profound impact. A crucial aspect of this improvement lies in accurately predicting future energy demands. Energy companies have been developing centralized demand forecasting models to achieve this goal. However, conventional centralized methods for developing such models face challenges concerning data privacy. This study proposes federated learning with a building attribute-based clustering approach to improve data privacy. The study establishes clusters based on hourly consumption data and building attributes, such as the space area of the building and the primary usage type of the building, considering the clustering based on hourly consumption data as the baseline approach. The results indicate that federated learning with attribute-based clustering outperforms federated learning without clustering for most clusters. Additionally, attribute-based clustering demonstrates promising potential as a privacy-preserving data clustering approach. Furthermore, integrating additional informative building attributes, such as window- to-wall ratio, compactness, and others, that are not included in the dataset used in this study could further enhance the accuracy of energy demand prediction using the proposed approach.