IMO-CoT: a benchmark from International Mathematics Olympiads for evaluating chain-of-thought reasoning in large language models
摘要
Large language models (LLMs), in recent times, have improved their problem-solving abilities, being able to solve complex mathematical problems with significant accuracies. Chain-of-thought reasoning is an innovative mechanism that allows LLMs to break down larger, complex problems into multiple sub-problems and into structured reasoning steps. This research introduces IMO-CoT, a novel, selective, information-rich benchmark derived from International Mathematics Olympiad (IMO) problems, designed to evaluate CoT reasoning capabilities in LLMs. Unlike existing reasoning datasets, IMO-CoT presents problems across Number Theory, Algebra, Combinatorics, and Geometry, each requiring multi-step reasoning with higher complexity than the other benchmarks. We have evaluated a diverse set of “proprietary” (closed) and “open” LLMs on two tasks—Direct Answering and Reasoning Continuation. It was observed experimentally that even the best of the LLMs were able to achieve only 9.22% accuracy (in the 2nd pass) for the direct answering task, and for reasoning continuation, the