In 2019, a new education curriculum was enacted in Taiwan, where the Information Technology (IT) education curriculum was also revamped. The focus of the new curriculum is computational thinking (CT) and problem-solving through programming. To understand students’ learning achievements, a new problem-solving assessment, the Chippy Challenge, was developed. Tasks in the challenge are taken from real-life scenarios, requiring students to use their problem-solving, analytical, and programming skills to solve. By analyzing the results from the Chippy Challenge, we can assess students’ CT and programming skills, and offer insights into students’ learning achievement. This paper examines and reports the change in students’ CT and programming skills before and after the new IT curriculum went into effect. This study included 9,735 participants, comparing achievements from 4,421 students to 5,314 students from before and after the new curriculum. Research methods include independent sample t-tests to analyze the differences in students’ CT and programming skills performance between the two groups. The results showed that students who went through the new curriculum had significantly better problem-solving skills compared to the other group. Furthermore, students from the urban schools performed significantly better than students from the rural schools. The results show that the new curriculum has a positive effect on students’ CT and programming problem-solving skills. However, the results are also alarming in that the majority of students still have lower-than-expected achievements.