This paper introduces the concept of a mixed grouping strategies for NOMA systems, analyzes its advantages over traditional OMA and fixed grouping strategies for NOMA systems, and proposes two feasible mixed grouping algorithms based on genetic algorithm and simulated annealing algorithm to improve system channel capacity. Through simulation verification, we find that the NOMA system with mixed grouping exhibits significant advantages in terms of system capacity and fairness. Furthermore, compared to brute force search algorithms, the proposed mixed grouping algorithms can significantly reduce computational complexity with only minor performance loss. Therefore, it can be concluded that the proposed mixed grouping algorithms strikes a good balance between performance and computational complexity. Additionally, this paper further explores the potential for implementing more complex NOMA system mixed grouping algorithms using machine learning methods in future scenarios.