In real-world scenarios, large language models (LLMs) can serve as assistants to help users accomplish their jobs and support the development of advanced applications. For the wide application of LLMs, inference efficiency is an essential concern that has been widely studied in existing work, accompanied by numerous optimization algorithms and code libraries to improve it. Nonetheless, users still find it challenging to compare the effectiveness of all the above methods and understand the underlying mechanisms. In this work, we propose a coarse-to-fine method that encompasses both experimental and analytical components. This method can be applied across various models and inference libraries. Specifically, we examine four usage scenarios within two practical applications. We further provide both theoretical and empirical fine-grained analyses of each module in the Transformer architecture. Our methods can be a general and invaluable resource for researchers to evaluate various code libraries and improve inference strategies across different LLMs. We open-source the supporting dataset, code, and evaluation scripts at the link: https://github.com/RUCAIBox/Inference-Efficiency-Evaluation .