Current Internet of Things (IoT) communication systems typically employ static protocols and hardcoded resource allocation mechanisms, limiting their ability to adapt to dynamic, resource-constrained environments. These schemes lack real-time adaptability and cannot provide fine-grained Quality of Service (QoS) management, particularly under network stress or time-varying conditions. This paper addresses this limitation by introducing QoSmart-IoT, a three-layered framework operating at client, cluster, and edge levels to enable real-time adaptation through QoS-driven decisions. The framework integrates hybrid clustering algorithms with adaptive Message Queuing Telemetry Transport (MQTT) and Constrained Application Protocol (CoAP) switching, utilizing live performance metrics including latency, energy consumption, throughput, and session-level security feedback. Balanced Iterative Reducing and Clustering using Hierarchies (BIRCH) and Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithms optimize resource allocation, while Advanced Encryption Standard (AES-128) provides security. The system demonstrates significant performance improvements, implemented on Contiki-NG and evaluated through comprehensive runtime analysis across multiple optimization scenarios and stress tests. Our work reduces average latency by 23.3% (from 55.1 ms to 42.3 ms), improves QoS scores from 0.82 to 0.88 during heavy load conditions, and achieves energy savings of up to 0.4 W during protocol switching operations. The adaptive clustering mechanism successfully executed 9 reconfigurations within 30 s each, enhancing recovery time and communication reliability to demonstrate QoSmart-IoT’s effectiveness in providing secure, adaptive performance in IoT environments.