Graph anomaly detection has been instrumental in many domains, playing critical roles in identifying unusual patterns in graph-structured data. Recent advancements that integrate contrastive learning with graph neural networks have demonstrated significant potential and achieved promising effectiveness results. However, these methods are usually time-consuming due to the expensive message propagation operations in graph neural networks. Therefore, how to design a method that is both effective and efficient remains a key challenge. In this paper, we present a new graph anomaly detection method TSGAD. Specifically, to keep it both effective and efficient, TSGAD adopts a two-stage design, consisting of contrastive pre-training and pseudo-label learning. The first stage simplifies graph neural networks into a pre-processing step and uses contrastive learning to make initial predictions. Considering the fact that supervised methods are usually more effective, the second stage builds an supervised learning model, based on the a few high-confidence predictions of the first stage. Extensive experimental results on four widely-studied datasets show both the effectiveness and efficiency of the proposed TSGAD. For example, as a fully unsupervised method that does not require any manual label annotations, TSGAD can even achieve comparable results with its supervised counterpart