REVIEW 25 cited by
LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present LLM-Blender, an ensembling framework designed to attain consistently superior performance by leveraging the diverse strengths of multiple open-source large language models (LLMs). Our framework consists of two modules: PairRanker and GenFuser, addressing the observation that optimal LLMs for different examples can significantly vary. PairRanker employs a specialized pairwise comparison method to distinguish subtle differences between candidate outputs. It jointly encodes the input text and a pair of candidates, using cross-attention encoders to determine the superior one. Our results demonstrate that PairRanker exhibits the highest correlation with ChatGPT-based ranking. Then, GenFuser aims to merge the top-ranked candidates, generating an improved output by capitalizing on their strengths and mitigating their weaknesses. To facilitate large-scale evaluation, we introduce a benchmark dataset, MixInstruct, which is a mixture of multiple instruction datasets featuring oracle pairwise comparisons. Our LLM-Blender significantly outperform individual LLMs and baseline methods across various metrics, establishing a substantial performance gap.
Forward citations
Cited by 25 Pith papers
-
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
Byte-Prefix Marginalization maps a teacher's next-token distribution onto the student's vocabulary through shared byte prefixes plus an explicit residual, giving a mass-preserving target for on-policy distillation acr...
-
How Good are LLM-based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Models
On a new benchmark of post-April 2025 queries, LLM rerankers show a 5-15% performance drop compared with familiar benchmarks, and lightweight models match them on efficiency and sometimes accuracy.
-
Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles
Diversity metrics used to select LLM ensembles are largely capability proxies; after control, only a modest pairwise co-failure association with majority-vote gain remains.
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
Opti-Q: A Constraint-Based Optimization Framework for Multi-LLM Question Planning
Per-question database-style plan search over multi-LLM DAGs improves QA quality under budgets by ~58% (MMLU-Pro) and ~41% (SimpleQA) versus reimplemented baselines.
-
When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines
In a 42-task controlled comparison, selecting the best candidate with judge panels beats MoA-style synthesis in every task, and a crossover threshold explains when team diversity helps.
-
StatEval: A Comprehensive Benchmark for Large Language Models in Statistics
StatEval is a new 16,000-question statistics benchmark showing that even strong LLMs score below 60% on research-level statistical proof tasks.
-
Dynamic Collaboration of Multi-Language Models based on Minimal Complete Semantic Units
MCSU-based vocabulary alignment plus distance-based dynamic selection (DDS) lets several LLMs vote token-by-token, beating single models and prior ensemble baselines on multiple reasoning benchmarks without training.
-
BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute
A routing system that chooses both the model and the number of samples per query to meet a quality threshold, yielding up to 60% cost savings.
-
AdaptiveLLM: A Framework for Selecting Optimal Cost-Efficient LLM for Code-Generation Based on CoT Length
AdaptiveLLM routes coding problems to cost-effective LLMs by clustering chain-of-thought lengths into three difficulty levels, improving pass@1 by 7.86% over ComplexityNet while cutting inference cost by 88.9%.
-
RewardAnything: Generalizable Principle-Following Reward Models
RewardAnything follows natural-language reward principles at inference time and, with the new RABench benchmark, demonstrates that principle-conditioned listwise training beats fixed-preference reward models on held-o...
-
How Much Backtracking is Enough? Exploring the Interplay of SFT and RL in Enhancing LLM Reasoning
On three controlled tasks, the optimal number of backtracks in SFT warm-up data grows with task difficulty, while trajectory correctness has little effect on final RL performance.
-
Universal Biological Sequence Reranking for Improved De Novo Peptide Sequencing
RankNovo, a list-wise reranker with mass-deviation supervision, improves de novo peptide sequencing accuracy by selecting among candidates from multiple base models.
-
A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation
Introduces a multi-role red teaming framework using attacker and jury models that increases attack success rates by up to 7.9% on LLM faithfulness in question-answering tasks.
-
Towards Generalized Routing: Model and Agent Orchestration for Adaptive and Efficient Inference
MoMA routes each query to a specialized agent or to the cost-optimal LLM, using judge-trained performance scores, a Pareto cost frontier, and TOPSIS selection.
-
How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMs
A leader LLM trained with a GRPO variant that conditions on frozen agent responses improves both collaborative and zero-shot accuracy on BBH, MATH, and MMLU.
-
MEMETRON: Metaheuristic Mechanisms for Test-time Response Optimization of Large Language Models
A memetic algorithm that applies genetic search and simulated annealing, with LLMs as the variation operators, to improve LLM responses with respect to an arbitrary reward function at inference time.
-
Leveraging LLMs to Evaluate Usefulness of Document
A cascade of LLM judges, fed with search context and behavior, produces multilevel usefulness labels for clicked documents and improves search satisfaction prediction.
-
Federated In-Context Learning: Iterative Refinement for Improved Answer Quality
Fed-ICL iteratively refines QA answers via federated in-context learning with only label transmission, showing convergence on a linear attention model and gains on MMLU and TruthfulQA.
-
LightRouter: Towards Efficient LLM Collaboration with Minimal Overhead
LightRouter uses short preview outputs to filter a pool of LLMs down to two, then aggregates their full responses, beating ensemble baselines and matching costlier models.
-
Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes
Statistical classifiers built on LLM activation norms and coordinates match or beat trained MLP heads on coarse intent routing and resist camouflage better, while MLPs win on fine-grained subfield distinctions.
-
DRF: LLM-AGENT Dynamic Reputation Filtering Framework
DRF combines an LLM rating network, a reputation update rule, and a UCB-style selection strategy to filter low-quality LLM agents during multi-agent task execution, reporting improved pass@1 and lower simulated cost o...
-
Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques
A survey of LLM routing and hierarchical inference techniques that proposes an unvalidated unified evaluation metric called the Inference Efficiency Score.
-
Bayesian Optimization for Enhanced Language Models: Optimizing Acquisition Functions
A bilevel Bayesian optimization strategy pairing EI and UCB acquisition functions improves RoBERTa fine-tuning by up to 2.7 percent on a subset of GLUE tasks, but the results lack statistical validation.
-
ORI: O Routing Intelligence
ORI routes queries by embedding cluster to the best model for the cluster's dominant benchmark, reporting modest gains that are not supported by its own routing rule or evaluation protocol.
Discussion (0). Continue with ORCID to comment.