Blockchain technology has facilitated the rapid advancement of smart contracts, whose programmability effectively supports practical business collaboration. However, security vulnerabilities in smart contracts—arising from their involvement in digital currency transactions, code immutability, and limitations inherent in the Solidity programming language—have led to significant security risks and substantial financial losses. To enhance the accuracy of vulnerability detection, this study proposes a machine learning–based detection model. Theoretical foundations of recurrent neural networks, autoencoder neural networks, and deep learning attention mechanisms are systematically presented. Through problem modeling, the importance of opcode graph structure generation and subspace mapping is clarified. A comprehensive algorithmic framework comprising four modules—feature preprocessing, feature mapping, feature extraction, and vulnerability detection—is designed. Experimental results demonstrate that Principal Component Analysis (PCA) achieves superior performance compared to Autoencoders (AE) in dimensionality reduction, while Graph Attention Networks (GAT) exhibit higher sensitivity to feature inputs than Graph Convolutional Networks (GCN). Furthermore, the integration of opcode graph structural features with expert-defined source code rules significantly enhances detection performance and model stability. This research provides an effective technical approach for optimizing smart contract security detection.