Malicious nodes pose a significant threat to the security and reliability of mixnet-based anonymous communication networks. This paper proposes a novel design that, on one hand, uses reputation and receipt-based mechanisms to dynamically detect and isolate malicious nodes during the operation of the anonymous system. On the other hand, the proposed cascade selection algorithm forces adversaries to choose between reducing the proportion of malicious behavior by their nodes and lowering the probability of generating a fully-malicious cascade. However, experimental simulation results indicate that, regardless of how adversaries balance these factors, the reliability of the anonymous network and the security of the cascades are effectively enhanced. Moreover, the proposed design can not only be used to improve the reliability of existing anonymous communication systems but also serve as a reference for designers of future anonymous communication systems.