The impressive capabilities of Large Language Models (LLMs) have opened new opportunities for constructing intelligent personal assistants (IPAs). People want to build IPAs that not only possess general abilities like translation and summarization but also the capability to use external APIs, ideally operating entirely on personal devices. Existing approaches usually rely on full fine-tuning to enhance LLMs with API calling abilities, which may lead to catastrophic forgetting of the foundation LLM. Methods of using multiple-step prompts without fine-tuning are limited by the performance of the selected APIs, and methods like vector retrieval often rely on powerful encoders deployed in the cloud, making them unsuitable for entirely offline operations. In this paper, we introduce EasIPA, which can enhance the ability to select APIs of LLMs on resource-constrained personal devices. By utilizing several tokens of the LLM, we improve the performance of API selection, allowing LLaMA 8B and its quantized version of LLMs to achieve comparable performance to GPT-3.5. Further robustness analysis indicates EasIPA’s robustness against different quantification methods and precisions, and ablation studies confirm the effectiveness of multi-layer knowledge injection.