Using lower-precision number formats can reduce the memory footprint, execution time and power consumption of machine learning algorithms and are therefore a key enabler for deployment of ML algorithms on constrained computing devices. Unfortunately it is non-trivial to predict the benefit of a smaller number format in a given model due to the complexity of modern machine learning frameworks, compilers and instruction set architectures. To address this situation, we suggest and implement a flexible end-to-end benchmarking workflow based on MLIR and LLVM. We demonstrate the workflow by generating code for a dot-product using three floating-point and twelve integer number formats. Five different MLIR/LLVM versions are used to generate code which is executed on an ARM Cortex A-72. We report generated LLVM and assembly code as well as actual execution times. We identify situations where smaller number formats yield lower performance as well as a major improvement for i8 throughput from MLIR 15.0.7 on.