The development of Large Language Models has known a rapid increase in term of research, applications and popularity where a large numbers of models with millions to billions of parameters, but this advancement was met with concerns in term of computation need, energy consumption and CO2 emission, which affect not only price but also the environment due to carbon footprint. This paper examines in detail how various state-of-the-art LLMs train on a single Graphical Processing Unit (GPU), paying close attention to crucial elements like throughput, memory utilization, training time and energy consumption. We find important trade-offs between model size, batch size and computational efficiency through empirical evaluation, offering practical advice for streamlining fine-tuning processes in the face of hardware constraints.