Recent advancements in large language models (LLMs) have enabled impressive capabilities across general-purpose tasks. However, applying these models to low-resource domains such as public administrative services in Vietnam remains a challenge due to data scarcity, domain complexity, and computational constraints. To address this, we present VietPSLM (Vietnamese Public Service Language Model), a compact, instruction-following language model fine-tuned for Vietnamese public service question answering. Our approach includes domain-adaptive pretraining, supervised fine-tuning, and a two-pass inference strategy to improve clarity and factual accuracy. Alongside VietPSLM, we release a suite of domain-specific datasets, including an unlabeled corpus for pretraining, a factual QA dataset, and two evaluation benchmarks. Despite its small size, VietPSLM delivers competitive performance, approaching the accuracy of larger proprietary systems such as Gemini 2.0 Flash. These results highlight that targeted adaptation and high-quality data can enable lightweight models to perform effectively in real-world governmental settings.