Scholarlink

<p>Verifying that a proposed solution truly resolves a scientific problem is central to trustworthy reasoning and retrieval. Using SCP-116K, we build 177,836 balanced problem–solution pairs (88,918 matched, 88,918 mismatched) spanning diverse STEM disciplines, and frame verification, following TRIZ/IDM, as distinguishing matched from mismatched pairs. Comparing lexical, retrieval-style, and lightweight neural models, our best model (RoBERTa + Slim ResNet, frozen sentence embeddings scored by a residual MLP) reaches an AUC of 0.966, an F1 of 0.905, and a LogLoss of 0.238. A CPU-friendly TF–IDF + Cosine + Elastic-Net baseline trails by 1.6–1.7 AUC points yet runs roughly 250<InlineEquation ID="IEq1"><EquationSource Format="TEX">\(\times\)</EquationSource></InlineEquation> faster in about 1.5&#xa0;GB of RAM, a strong efficiency–accuracy trade-off. The probabilities act as re-ranking scores over candidate solutions; we read the high ROC–AUC as pairwise discrimination and absolute accuracy as an upper bound given the synthetic negatives.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A large-scale benchmark shows lightweight models can distinguish matched from mismatched problem–solution pairs across diverse STEM disciplines

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月1日

查看全文
  • Nicolas Douard,
  • Ahmed Samet,
  • George Giakos,
  • Denis Cavallucci

摘要

Verifying that a proposed solution truly resolves a scientific problem is central to trustworthy reasoning and retrieval. Using SCP-116K, we build 177,836 balanced problem–solution pairs (88,918 matched, 88,918 mismatched) spanning diverse STEM disciplines, and frame verification, following TRIZ/IDM, as distinguishing matched from mismatched pairs. Comparing lexical, retrieval-style, and lightweight neural models, our best model (RoBERTa + Slim ResNet, frozen sentence embeddings scored by a residual MLP) reaches an AUC of 0.966, an F1 of 0.905, and a LogLoss of 0.238. A CPU-friendly TF–IDF + Cosine + Elastic-Net baseline trails by 1.6–1.7 AUC points yet runs roughly 250\(\times\) faster in about 1.5 GB of RAM, a strong efficiency–accuracy trade-off. The probabilities act as re-ranking scores over candidate solutions; we read the high ROC–AUC as pairwise discrimination and absolute accuracy as an upper bound given the synthetic negatives.