Deploying Large Language Models (LLMs) in offline educational contexts, particularly on edge devices like smartphones, presents significant computational challenges due to their high resource demands. Addressing this issue, AIED Unplugged proposes using lightweight LLMs optimized for mobile platforms, yet understanding the practical performance-quality trade-offs remains limited. This study investigates these trade-offs through benchmarking six compact LLMs embedded via the Llama.cpp framework into an Android application on a low-resource smartphone. Performance metrics (response time, memory usage, and storage size) were tracked using Sentry, while DeepEval assessed response quality based on relevance, clarity, accuracy, and completeness. The evaluation demonstrated notable trade-offs: lighter models, such as MobileVLM and TinyLlama, exhibited fast inference speeds but lower quality in completeness and accuracy, whereas slightly larger models like Qwen2 and DeepSeek provided enhanced accuracy and completeness but suffered from slower responses and increased memory requirements. These insights underscore the necessity of targeted optimization strategies, including model quantization and hybrid deployments, to effectively balance resource efficiency and educational quality. Ultimately, this research supports equitable access to learning materials for marginalized people by providing insights for AI-powered education in offline situations.