Introduction The significance of Physical Education (PE) in promoting healthy habits and encouraging active lifestyles among students and youth can positively influence their confidence in physical activities while lowering participation anxiety. Purpose The study examined the expectancy beliefs, task values and anxiety of 480 primary school pupils (234 males and 246 females) aged between 10 and 12 years (mean age: 1.96 ± 0.82) from Year 4 to Year 6 in Kuching, Sarawak. Methods The Expectancy-Value Model Questionnaire and Physical Education Classroom Anxiety Scale were utilised to examine the pupils’ expectancy beliefs, task values (attainment, utility, intrinsic value, cost) and anxiety towards physical activity during PE classes. Descriptive analysis, independent-samples t-test, one- and two-way ANOVA were applied on data. Results The independent-samples t-test revealed significant differences between males and females in expectancy beliefs (p = 0.001), attainment (p = 0.007), utility (p = 0.002), intrinsic value (p = 0.045), fear of negative evaluation (p = 0.030), and test anxiety (p = 0.004). Furthermore, the one-way ANOVA revealed significant differences between the three age groups in expectancy beliefs (p = 0.045), utility (p = 0.019), and cost (p = 0.009). Moreover, the between-subjects ANOVA revealed significant main effects of gender on expectancy beliefs, attainment, utility, fear of negative evaluation, and test anxiety (p < 0.05). Additionally, results showed significant main effects of age on expectancy beliefs, utility, and cost (p < 0.05). There were also significant interactions between gender and age on cost (p = 0.006), communication apprehension (p = 0.028), and fear of negative evaluation (p = 0.008). Conclusion The study’s findings suggest that physical activity interventions that prioritize beliefs, task values, and cost reduction can lead to reduced anxiety, improved performance, and long-term sustainability.