In order to explore the mathematical algorithm optimization strategy of computer programming, this paper combines the case analysis method to carry out the analysis, and aiming at various problems and redundant coding in traditional programming, this paper puts forward a strategy model of automatically defining function modules. In this method, the neural network is used to automatically extract the repetitive structures, and form function modules that can call each other, so as to get the final complex expression by nesting. Moreover, the network of extracting function module and the network of generating module scheduling are separated, and the parameter optimization is carried out by co-evolution, which completes the independence of these two different process networks and further reduces the optimization dimension. Finally, a lot of tests and analysis are carried out on classic benchmark data sets, and it is found that the model corresponding to this strategy shows excellent results in all performances.