The transcription of historical manuscripts presents unique challenges due to complex writing styles, pervasive abbreviations, and significant linguistic heterogeneity. Recent multimodal architectures that integrate powerful language models with vision encoders offer promising and flexible solutions. In this work, we propose a two-stage training strategy to model two main transcription paradigms: diplomatic (abbreviated) and semi-diplomatic (expanded). Our approach leverages two specialized corpora—CATMuS Medieval (abbreviated) and TRIDIS (non-abbreviated). In Stage 1, we pre-train a single model on a random combined dataset using a loss penalization mechanism to discourage the prediction of Medieval Unicode Font Initiative (MUFI) characters during expanded-text generation. In Stage 2, this pre-trained model serves as the foundation to train separate LoRA adapters for each transcription style, which can be dynamically switched at inference. We benchmark our approach using MiniCPM-Llama3-V-2.5, Phi-3.5 Vision and Qwen2-VL (all published end-2024) against two established baselines (Kraken v5 and TrOCR). Our results indicate that while the unified “double-head” model provides a solid initialization, fine-tuning separate, switchable LoRA adapters leveraging that base yields progressive superior performance by better adapting to each transcription style. We discuss the limitations of a single adapter and advocate for switched adapters to achieve optimal flexibility. This integrated approach supports historians, linguists, and digital humanities scholars in exploring complex historical corpora at multiple interpretive levels.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Dual-Style Transcription of Historical Manuscripts Based on Multimodal Small Language Models with Switchable Adapters

  • Sergio Torres Aguilar

摘要

The transcription of historical manuscripts presents unique challenges due to complex writing styles, pervasive abbreviations, and significant linguistic heterogeneity. Recent multimodal architectures that integrate powerful language models with vision encoders offer promising and flexible solutions. In this work, we propose a two-stage training strategy to model two main transcription paradigms: diplomatic (abbreviated) and semi-diplomatic (expanded). Our approach leverages two specialized corpora—CATMuS Medieval (abbreviated) and TRIDIS (non-abbreviated). In Stage 1, we pre-train a single model on a random combined dataset using a loss penalization mechanism to discourage the prediction of Medieval Unicode Font Initiative (MUFI) characters during expanded-text generation. In Stage 2, this pre-trained model serves as the foundation to train separate LoRA adapters for each transcription style, which can be dynamically switched at inference. We benchmark our approach using MiniCPM-Llama3-V-2.5, Phi-3.5 Vision and Qwen2-VL (all published end-2024) against two established baselines (Kraken v5 and TrOCR). Our results indicate that while the unified “double-head” model provides a solid initialization, fine-tuning separate, switchable LoRA adapters leveraging that base yields progressive superior performance by better adapting to each transcription style. We discuss the limitations of a single adapter and advocate for switched adapters to achieve optimal flexibility. This integrated approach supports historians, linguists, and digital humanities scholars in exploring complex historical corpora at multiple interpretive levels.