Large language models (LLMs) are widely used in various fields, and the demand for LLM inference on mobile devices is also growing. LoRA technology, due to its high efficiency and low cost, is a basic method for fine-tuning pre-trained models for downstream tasks. However, due to the limited hardware resources of mobile devices, ensuring efficient inference while fully utilizing mobile hardware resources becomes a key issue. Considering the characteristics of heterogeneous computing resources and unified memory architecture on mobile devices, we propose MHP-SERVE, an LLM inference framework for mobile devices. MHP-SERVE ensures effective partitioning of LoRA-based models, guaranteeing that LoRA inference computations run on the CPU and base model computations run on the GPU. It also designs a heterogeneous parallel execution scheme to effectively ensure the correct sequential execution of synchronization points. By using shared memory in a unified memory architecture to share tensors, it optimizes the delay caused by memory copies in existing solutions, achieving efficient CPU-GPU data transfer. Extensive experiments show that this framework reduces the latency of generating the first token by 10%-20% and the overall end-to-end inference by 10%.