Effective code pre-processing model for code search
摘要
Against the backdrop of advancing code search research, the impact of source code preprocessing on code search accuracy has become increasingly pronounced, making the development of efficient source code preprocessing methods a critical endeavor in this field. In this paper, we propose JSTTR (Joint AST Statement Tree and Token Common Representation), a novel code preprocessing model–note that to our knowledge, this marks the first integration of AST statement trees into code search research. JSTTR leverages the ST-Tree (AST Statement Tree) and token-based representation to jointly capture syntactic and semantic information from source code, respectively, thereby enhancing the characterization of source code features and improving code search accuracy. To validate JSTTR’s efficacy, we conducted comparative and ablation experiments against baseline models (Token+Path, Token+SBT, ASTNN), and the results demonstrate its superior performance: compared to these baselines, JSTTR achieves improvements of 6.45%, 3.12%, and 3.18% in MRR (Mean Reciprocal Rank); reductions of 2.03%, 24.83%, and 35.51% in NDCG (Normalized Discounted Cumulative Gain); increases of 1.52%, 0.65%, and 1.24% in ACC (Accuracy); boosts of 6.04%, 1.23%, and 4.36% in R@1 (Recall@1); gains of 2.60%, 0.80%, and 2.04% in R@5 (Recall@5); and enhancements of 1.41%, 0.55%, and 1.15% in R@10 (Recall@10). Collectively, these results confirm that JSTTR yields stronger code search outcomes and optimal internal component synergy, not only effectively enhancing code search accuracy but also establishing a promising research direction for source code preprocessing in code search.