Tamil is an agglutinative and highly inflectional language with rich morphology. Sandhi errors are a class of spelling mistakes in Tamil that occur at word boundaries due to incorrect insertion or omission of hard consonants. This paper presents TamilSandhi, an open-source neuro-symbolic framework for identifying and correcting such errors. It combines rule-based logic for simple Sandhi rules with neural models for one complex rule. The rule-based system implements 12 Vallinam (hard consonant) addition rules and 8 deletion rules, derived from classical Tamil grammar texts such as Tolkappiyam and Nannool. These rules were packaged into a PyPi library ( https://pypi.org/project/tamilsandhi-toolkit/ ) and validated using a suite of 300 unit tests (235 for addition, 65 for deletion). To address a complex rule not covered by the rule-based logic, a neural sequence-to-sequence approach was used. A supervised corpus of 10,434 manually annotated sentence pairs, each consisting of an incorrect and corrected version, was created. Three multilingual transformer models—mBART, mT5, and NLLB—were fine-tuned. Among them, mBART achieved the best performance, with a BLEU score of 99.9 and exact match accuracy of 97.9%.TamilSandhi is released as an open-source project on GitHub ( https://github.com/TamilGeekGirl/TamilSandhiNeuroSymbolicAI/ ). Due to its modularity, reproducibility, and linguistic validity, TamilSandhi constitutes a significant contribution to NLP research in a widely used but low-resource language.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

TamilSandhi: A Neuro-Symbolic AI Toolkit for Correcting Sandhi Errors in Tamil

  • V. M. Yazhmozhi,
  • Annalu Waller,
  • Jacky Visser

摘要

Tamil is an agglutinative and highly inflectional language with rich morphology. Sandhi errors are a class of spelling mistakes in Tamil that occur at word boundaries due to incorrect insertion or omission of hard consonants. This paper presents TamilSandhi, an open-source neuro-symbolic framework for identifying and correcting such errors. It combines rule-based logic for simple Sandhi rules with neural models for one complex rule. The rule-based system implements 12 Vallinam (hard consonant) addition rules and 8 deletion rules, derived from classical Tamil grammar texts such as Tolkappiyam and Nannool. These rules were packaged into a PyPi library ( https://pypi.org/project/tamilsandhi-toolkit/ ) and validated using a suite of 300 unit tests (235 for addition, 65 for deletion). To address a complex rule not covered by the rule-based logic, a neural sequence-to-sequence approach was used. A supervised corpus of 10,434 manually annotated sentence pairs, each consisting of an incorrect and corrected version, was created. Three multilingual transformer models—mBART, mT5, and NLLB—were fine-tuned. Among them, mBART achieved the best performance, with a BLEU score of 99.9 and exact match accuracy of 97.9%.TamilSandhi is released as an open-source project on GitHub ( https://github.com/TamilGeekGirl/TamilSandhiNeuroSymbolicAI/ ). Due to its modularity, reproducibility, and linguistic validity, TamilSandhi constitutes a significant contribution to NLP research in a widely used but low-resource language.