Scholarlink

<p>Large Language Models (LLMs) differ widely in their performance across tasks, making efficient model selection essential for reliable and cost-effective deployment. This paper proposes a zero-shot LLM ranking framework that predicts the most suitable model for a given prompt without executing any candidate models. Using data from the TREC Million LLM Track, which includes 14,950 prompts evaluated across 1130 LLMs, the framework integrates prompt-aware, cluster-aware, and LLM metadata-aware embeddings within an end-to-end neural architecture. The proposed model achieved an nDCG@10 of 0.3451 and an MRR of 0.2550, representing a 38% improvement over single-feature baselines. Analysis across 2,990 test prompts showed that ranking effectiveness varies with prompt type, length and prompt search intent. The results demonstrate that fusing heterogeneous features enables accurate zero-shot LLM selection while significantly reducing computational cost. This work provides a scalable and energy-efficient alternative to brute-force evaluation and establishes a foundation for adaptive, prompt-aware routing in multi-LLM systems.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A zero-shot neural learning to rank framework for ranking large language models

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年2月18日

查看全文
  • Shariq Bashir,
  • Mohammad Alkhatib

摘要

Large Language Models (LLMs) differ widely in their performance across tasks, making efficient model selection essential for reliable and cost-effective deployment. This paper proposes a zero-shot LLM ranking framework that predicts the most suitable model for a given prompt without executing any candidate models. Using data from the TREC Million LLM Track, which includes 14,950 prompts evaluated across 1130 LLMs, the framework integrates prompt-aware, cluster-aware, and LLM metadata-aware embeddings within an end-to-end neural architecture. The proposed model achieved an nDCG@10 of 0.3451 and an MRR of 0.2550, representing a 38% improvement over single-feature baselines. Analysis across 2,990 test prompts showed that ranking effectiveness varies with prompt type, length and prompt search intent. The results demonstrate that fusing heterogeneous features enables accurate zero-shot LLM selection while significantly reducing computational cost. This work provides a scalable and energy-efficient alternative to brute-force evaluation and establishes a foundation for adaptive, prompt-aware routing in multi-LLM systems.