As computational needs expand, new computing paradigms such as GPUs, FPGAs, high-performance computing clusters, digital annealers, neuromorphic computing systems, and quantum computers are emerging to complement traditional CPU-based computing models. Each paradigm offers unique capabilities for combinatorial optimization, a field concerned with finding the best solution from a finite set of possibilities. This paper addresses the challenge of fairly benchmarking the performance of combinatorial optimization solvers across these diverse paradigms. We propose a holistic approach to benchmarking that includes recommendations for fair comparisons and the introduction of new metrics. Our findings highlight the need for clear and equitable comparison criteria, particularly when contrasting digital and analogue platforms or different algorithm classes.