The rapid advancement of large AI models such as GPT-4 has transformed numerous domains. However, their deployment on edge devices remains limited by resource constraints, prompting the need for collaborative training with lightweight models. Existing methods often rely on fixed data granularities, which hinder adaptation to models with diverse capacities. Meanwhile, data fragmentation and privacy concerns in edge–cloud systems hinder the centralized construction of multi-granularity representations. Unsupervised methods further struggle to cope with high-dimensional complexity and diverse Quality of Service (QoS) demands. In this paper, we propose Cultivator, a QoS-aware framework for the distributed construction of multi-granularity trees, designed for heterogeneous model scales. Cultivator integrates multi-modal fusion with Federated Learning (FL) to address fragmented data and limited semantics. A dynamic QoS-aware mechanism further guides strategy selection between balanced regularized consistency for low-latency needs and optimal-transport-based multi-modal consistency for high-performance scenarios. Evaluations on various datasets demonstrate that Cultivator improves overall accuracy by up to 11.91% and increases accuracy density by an average of 13.64%, highlighting its efficiency in enhancing distributed training under complex edge–cloud environments.