Scholarlink

<p>Transformer-based sequence-generation models have achieved remarkable success across a range of NLP tasks. However, ensuring they adhere to complex, domain-specific constraints remains a major challenge—especially in settings like academic planning and workflow automation. In this paper, we introduce a constraint-based Transformer that promotes rule-adherence via bias injection. During training, each input sequence is augmented with canonical, rule-compliant examples whose positional embeddings are integrated into the model’s attention bias matrices, steering the Transformer toward valid orderings without sacrificing generalization. We demonstrate this approach on an academic-planning task, evaluating performance on a held-out synthetic dataset using a constraint-adherence metric (the percentage of outputs satisfying all domain rules) alongside standard language model metrics. Our model achieves 87.40% adherence—surpassing a standard Transformer (87.10%) and an LSTM model (85.50%)—while matching their perplexities (1.28) and Top-5 accuracies (98.0%). These results show that bias injection can effectively bridge rule-based and data-driven methods, delivering more reliable recommendations. Future work will explore adaptive bias schedules and extensions to heterogeneous rule sets.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Constraint-biased transformers: attention bias injection for rule-compliant course recommendation

  • 期刊论文

  • 发表日期：

    2026年3月11日

查看全文
  • Ahmed Ibrahim

摘要

Transformer-based sequence-generation models have achieved remarkable success across a range of NLP tasks. However, ensuring they adhere to complex, domain-specific constraints remains a major challenge—especially in settings like academic planning and workflow automation. In this paper, we introduce a constraint-based Transformer that promotes rule-adherence via bias injection. During training, each input sequence is augmented with canonical, rule-compliant examples whose positional embeddings are integrated into the model’s attention bias matrices, steering the Transformer toward valid orderings without sacrificing generalization. We demonstrate this approach on an academic-planning task, evaluating performance on a held-out synthetic dataset using a constraint-adherence metric (the percentage of outputs satisfying all domain rules) alongside standard language model metrics. Our model achieves 87.40% adherence—surpassing a standard Transformer (87.10%) and an LSTM model (85.50%)—while matching their perplexities (1.28) and Top-5 accuracies (98.0%). These results show that bias injection can effectively bridge rule-based and data-driven methods, delivering more reliable recommendations. Future work will explore adaptive bias schedules and extensions to heterogeneous rule sets.