Validation of Short Answers for Open Questions from the FairytaleQA-Sp Corpus
摘要
Short-response grading is a central issue in reading comprehension evaluation. In this regard, this work first introduces an enriched dataset with both human-written and artificial intelligence–generated responses, and then compares transformer-based models, similarity metrics, and large language models for automatic validation of open-ended responses. The fine-tuned RoBERTa binary classifier achieved competitive performance, but DeepSeek-V3 outperformed all models, including ChatGPT-4o. A Sentence Transformer model trained with contrastive learning showed limitations in detecting incorrect answers. We discuss the strengths and weaknesses of each approach and propose hybrid models that are better aligned with pedagogical goals.