Mobile edge computing (MEC) is a promising paradigm that brings communication and computing resources closer to mobile users, thereby enabling low-latency and high-quality services. However, the low energy efficiency of edge servers (ESs) remains a critical issue, as they are often kept continuously active to handle irregular and unpredictable task arrivals. In fact, tasks can be offloaded from one ES to another ES or to the cloud, so that the idle ESs can be sleep down to save energy. Unfortunately, existing work often ignores the delay caused by the sleep operations of ESs, which limits their applicability in real-world scenarios. In this paper, we investigate the problem of energy-efficient task offloading in MEC-cloud. Tasks can be dynamically offloaded either among ESs or from ESs to the cloud, depending on the system state. To achieve energy savings, we consider both wake-up and sleep-down delays of the ESs. To address this problem, we first formulate it as a Markov Decision Process (MDP) and propose a Soft Actor-Critic (SAC)-based algorithm to dynamically offload the tasks from ESs to ESs/cloud. Then, we design a heuristic algorithm to wake up or sleep down the ESs for energy saving. Simulation results demonstrate that our method significantly reduce the time-average energy consumption of ESs as well as the time-average transmission and computation delay of all tasks.