Open-ended mathematical problems are commonly used to assess students’ abilities, during which teachers provide comments in addition to scores, offering valuable procedural and conceptual support, while this process lacks automation. Prior automated assessments primarily used natural language processing, focusing on students’ textual answers. However, open-ended mathematical problems often require answers to include images, such as number lines, geometric shapes, and charts. Yet, limited methods are available for the automated scoring of image-based responses, and even fewer multi-modal approaches can simultaneously process both text and images. This study builds on prior research of automated assessment of open-ended math questions, developing and evaluating a multi-modal framework capable of predicting scores. Using historical log data and leveraging textual and visual artifacts from students, we utilized sample-level multi-modal semantic representations generated by generative artificial intelligence (GAI) to propose a collaborative filtering-based machine learning approach for predicting student scores. Our research demonstrates that incorporating predicted scores similar to historical answers can significantly enhance the performance of the scoring model. This paper provides a fresh perspective and technological framework for multi-modal technology in automatic feedback system. Knowledge tracing aims to model learners’ knowledge states based on their problem-solving sequences. However, most existing methods fail to incorporate rich learning process data (auxiliary variables), which characterizes a multi-feature scenario. To address this limitation, we propose a plug-and-play multi-feature encoding framework that can be integrated into the embedding layer of deep knowledge tracing (DKT) for joint training, enabling effective representation learning of auxiliary variables. Our framework first selects the top-k features using time-series-based conditional mutual information and then employs a stacked transformer-encoder to extract feature representations, structured as embeddings across different channels. Given that different features may have varying influences on learners’ knowledge states, we design a stacked multi-head attention mechanism to assign distinct weights to different channel-time combinations. Using a real-world dataset collected from a middle school mathematics learning platform, we demonstrate that the proposed method effectively encodes auxiliary variables, enhancing both the performance and interpretability of DKT.