The exceptional performance of large language models drives their rapid development and constantly expanding range of applications. This growth increases the demand for fine-tuning open-source models to suit specific use cases. However, fine-tuning LLMs requires advanced expertise in deep learning tools and distributed computing. Deploying them for evaluation can also be complex and resource-intensive. To address these challenges, we built LLM ProtoHub, a system designed to simplify and streamline the fine-tuning and deployment of large language models. LLM ProtoHub abstracts the details of distributed training, model versioning, and deployment, enabling users to focus on experimentation and innovation. The system leverages low-rank adapters (LoRA), significantly reducing disk space requirements and enabling efficient GPU resource utilization by serving multiple versions of the base model on a single GPU. The experiments prove that our approach allows for effective LLM fine-tuning with minimal performance overhead. Additionally, we showcase two applications developed using LLM ProtoHub, illustrating its versatility in creating LLM-based solutions. LLM ProtoHub was developed as part of the CAISE platform, a local cloud computing environment, and was deployed at the CI TASK supercomputer center.