In response to the suboptimal obstacle avoidance performance of Autonomous Underwater Vehicle clusters (AUV clusters) and the issues of slow convergence speed and poor network performance associated with MATD3 networks, we propose a novel AUV cluster obstacle avoidance algorithm based on DHMATD3. By computing the sliding window average of the iterative TD error during the MATD3 network training process, we dynamically adjust the delayed update frequency of the MATD3 network. Subsequently, we utilize a GRU network to process historical sample data and incorporate it into the experience pool, thereby altering the sampling method of the MATD3 network experience pool to expedite the learning of excellent data. Given that certain outputs of the GRU network may contain irrelevant information, we introduce an attention mechanism to ensure that the GRU network focuses solely on useful information, thereby optimizing the structure of the MATD3 network. We apply this approach to address AUV cluster obstacle avoidance problems. Experimental results demonstrate that this algorithm effectively enhances the performance and convergence speed of the MATD3 network, leading to more optimal AUV cluster obstacle avoidance decisions and increased stability.