The exponential proliferation of Internet of Things (IoT) devices and the emergence of latency-critical applications have created unprecedented demands for intelligent task offloading mechanisms in heterogeneous edge-cloud computing paradigms. Existing approaches predominantly rely on oversimplified bi-level optimization frameworks that inadequately capture the computational heterogeneity and dynamic resource variations inherent in real-world edge deployments. This paper proposes HDMARL, a novel Hierarchical Deep Multi-Agent Reinforcement Learning framework that systematically addresses task offloading optimization across heterogeneous computational tiers. Our approach introduces a three-tier decision hierarchy encompassing local IoT processing, capability-aware edge server selection, and adaptive cloud resource allocation. The framework employs specialized Deep Q-Network (DQN) agents for device-level decisions and Actor-Critic mechanisms for global coordination, integrated with a dynamic priority-aware scheduling algorithm that jointly considers task urgency and resource heterogeneity. Comprehensive experimental evaluation on realistic IoT deployment scenarios demonstrates that HDMARL achieves superior performance with a 23.4% reduction in average task completion latency, 18.7% improvement in energy efficiency, and 31.2% enhancement in resource utilization compared to state-of-the-art approaches, while maintaining logarithmic scalability characteristics.