SCoT Guard: A Safety Chain of Thought Guardrail Model
摘要
The proliferation of large language models (LLMs) in intelligent applications has introduced critical safety challenges concerning input and output content moderation. Current guardrail solutions suffer from inadequate performance in small-parameter models while excessive resource overhead in large-parameter models, limited capacity for fine-grained rule customization, and improvable out-of-distribution generalization. To overcome these limitations, we introduce SCoT Guard, a novel reasoning-based guardrail framework that dynamically acquires fine-grained judgment criteria through contextual learning from examples. Our approach leverages a meticulously curated training dataset of 40,000 high-quality instances generated using DeepSeek-R1, incorporating comprehensive safety detection reasoning chains and rule-learning mechanisms. We implement LoRA fine-tuning on the Qwen2.5-1.5B-Instruct model to achieve optimal performance. At inference time, SCoT Guard utilizes Retrieval-Augmented Generation (RAG) to identify similar examples, extracts their underlying judgment principles, and applies these learned rules to evaluate the new sample, thereby accommodating evolving safety requirements. Comprehensive experimental evaluation demonstrates that SCoT Guard substantially surpasses 9 baseline methods across 6 benchmarks (e.g., OpenAI Moderation, ToxicChat, BeaverTails). These results confirm the effectiveness of integrating chain-of-thought reasoning with RAG in enhancing the performance, interpretability, and adaptability of guardrail models. With its compact 1.5B parameter architecture, SCoT Guard enables direct deployment on resource-limited IoT edge devices, facilitating real-time local inference while eliminating cloud dependency and ensuring data privacy. Both the training dataset and model weights will be open-sourced.