With the development of wireless communication systems, human-oriented mobile communication networks are gradually evolving to heterogeneous networks with human-machine-object hybrid access. Due to the heterogeneous nature of the user devices and the complexity of the network parameters, energy efficiency, a key metric for evaluating both the data transmission and the network survivability, becomes difficult to optimize in this scenario. In this paper, we study the energy efficiency optimization problem based on deep reinforcement learning (DRL) in a heterogeneous network with human-machine-object hybrid access, which contains unmanned aerial vehicle (UAV), reconfigurable intelligent surface (RIS), macro base station (MBS), small base station (SBS), object-type users, human-type users, and MBS user equipments (MUEs). A distributed multi-agent assisted resource allocation algorithm based on dueling double deep Q network (D3QN), termed DMARA-D3QN, is proposed to optimize the user transmit power allocation, channel allocation, RIS phase shifts, and user association factors to maximize the system energy efficiency. Simulation results show that the proposed algorithm converges quickly and stably compared to other DRL algorithms. Compared to the amplify-and-forward (AF) relay, the RIS can offer a 25.3% boost in energy efficiency. Moreover, the proposed algorithm can achieve at least a 37.5% energy efficiency improvement over other optimization strategies.