Log parsing is a critical step for log analysis, responsible for converting raw, semi-structured log messages into structured templates. Despite using semantic knowledge to enhance template extraction, LLM-based log parsers frequently generate fragmented templates, generating several marginally distinct templates for semantically identical events. This fragmentation generates duplicate downstream processing, reduces cache hit rates, and inflates the template pool. We introduce EM-Merge, a lightweight, model-independent postprocessing layer that directly mitigates fragmentation without requiring further LLM calls. EM-Merge employs a two-stage strategy: (1) semantic screening, where newly generated and cached templates are encoded with Sentence-BERT and nearest-neighbor search selects semantically plausible candidates; (2) confidence-weighted structural merging, where candidate pairs are scored by a normalized longest-common-subsequence measure modulated by LLM generation confidences, then clustered and merged into generalized templates (non-common tokens replaced by wildcards). This design preserves semantic distinctions while preventing misleading splits driven by punctuation or surface variation. We evaluate EM-Merge on 14 public log datasets and find that, with no runtime overhead, it reliably reduces template redundancy, improves grouping and parsing accuracy, and lowers end-to-end processing cost when compared to strong baselines.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

EM-Merge: Consolidating Fragmented Templates in LLM Log Parsing

  • Qiyue Zhu,
  • Jinyuan Wang,
  • Yani Yang,
  • Tong Li

摘要

Log parsing is a critical step for log analysis, responsible for converting raw, semi-structured log messages into structured templates. Despite using semantic knowledge to enhance template extraction, LLM-based log parsers frequently generate fragmented templates, generating several marginally distinct templates for semantically identical events. This fragmentation generates duplicate downstream processing, reduces cache hit rates, and inflates the template pool. We introduce EM-Merge, a lightweight, model-independent postprocessing layer that directly mitigates fragmentation without requiring further LLM calls. EM-Merge employs a two-stage strategy: (1) semantic screening, where newly generated and cached templates are encoded with Sentence-BERT and nearest-neighbor search selects semantically plausible candidates; (2) confidence-weighted structural merging, where candidate pairs are scored by a normalized longest-common-subsequence measure modulated by LLM generation confidences, then clustered and merged into generalized templates (non-common tokens replaced by wildcards). This design preserves semantic distinctions while preventing misleading splits driven by punctuation or surface variation. We evaluate EM-Merge on 14 public log datasets and find that, with no runtime overhead, it reliably reduces template redundancy, improves grouping and parsing accuracy, and lowers end-to-end processing cost when compared to strong baselines.