Programming syntax is highly complex, which has made it consistently difficult to automate feedback for compiler errors in educational settings. Large language models (LLMs) show promise for addressing this issue at scale by providing personalized feedback tailored to specific code submissions, but the effectiveness of this feedback remains uncertain. This study evaluated the impact of GPT-4o at generating real-time feedback for compiler errors during a randomized controlled trial. A total of 248 CS1 students participated, submitting 22,674 pieces of code to an automated programming assessment platform. Students in the Experimental group received LLM feedback, while the Control group did not. Results showed that students who received LLM feedback rated it highly for usefulness, and submitted fewer non-compiling code attempts. These students also had significantly improved performance in terms of resolving errors in consecutive attempts compared to the Control group. Affective surveys revealed that the LLM feedback group self-reported higher focus and lower levels of “confrustion” (a combination of confusion and frustration) after encountering compiler errors. When LLM feedback was temporarily disabled, students in the Experimental group solved programming problems more quickly and demonstrated significant improvement in resolving errors across attempts. However, no significant differences were observed between groups in terms of final scores on a simulated exam. These findings suggest that LLM-generated feedback can improve students’ coding experience, engagement, and problem-solving efficiency in the initial phases of computer science education, though it may not lead to better final performance.