The digital transition plays a critical role in decarbonizing building energy systems. However, the existing data information infrastructure, such as the semantic data model, has not been fully exploited in smart applications. This study introduces a novel two-layer approach that combines semantic data models with data-driven energy parameter prediction. The developed method uses a real-world data provided by property owners and operators of the apartment building, located in Stockholm, Sweden. This use case includes time series sensory data of a ventilation system and other meta data, such as the control cards. The core of the approach consists of two interconnected layers: a semantic data model, referred to as a knowledge graph, built using ontologies to organize and contextualize diverse data sources. The second layer consists of data-driven thermal parameter prediction models, offering predictive insights into system behavior. A Large Language Model (LLM) is developed and applied to integrate these two layers, enabling a human-in-the-loop interaction between the two layers. This integration allows intuitive querying of data and energy management. The key contribution of this paper is a novel approach to combine ontology-based semantic modeling with data-driven, creating a framework for future smart building control and management.