REVIEW 9 cited by
Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Many recent state-of-the-art results in language tasks were achieved using compound systems that perform multiple Language Model (LM) calls and aggregate their responses. However, there is little understanding of how the number of LM calls - e.g., when asking the LM to answer each question multiple times and taking a majority vote - affects such a compound system's performance. In this paper, we initiate the study of scaling properties of compound inference systems. We analyze, theoretically and empirically, how the number of LM calls affects the performance of Vote and Filter-Vote, two of the simplest compound system designs, which aggregate LM responses via majority voting, optionally applying LM filters. We find, surprisingly, that across multiple language tasks, the performance of both Vote and Filter-Vote can first increase but then decrease as a function of the number of LM calls. Our theoretical results suggest that this non-monotonicity is due to the diversity of query difficulties within a task: more LM calls lead to higher performance on "easy" queries, but lower performance on "hard" queries, and non-monotone behavior can emerge when a task contains both types of queries. This insight then allows us to compute, from a small number of samples, the number of LM calls that maximizes system performance, and define an analytical scaling model for both systems. Experiments show that our scaling model can accurately predict the performance of Vote and Filter-Vote systems and thus find the optimal number of LM calls to make.
Forward citations
Cited by 9 Pith papers
-
Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety
Map-reduce scaffolding degrades measured safety mainly by stripping multiple-choice options (40–89% of the loss is format conversion); scaffold architecture explains only 0.4% of variance and composite safety scores h...
-
DeepLook: Deeper Thinking with Lookahead
A training-free decoder that branches only at low-confidence reasoning segments and ranks continuations by average lookahead confidence reports equal or better accuracy at roughly 1/8 of a confidence-filtered baseline...
-
Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms
Murakkab uses declarative workflow specs and a profile-guided MILP optimizer to reduce GPU, energy, and cost for agentic workflow serving while meeting percentile-defined SLOs.
-
BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute
A routing system that chooses both the model and the number of samples per query to meet a quality threshold, yielding up to 60% cost savings.
-
TongSearch-QR: Reinforced Query Reasoning for Retrieval
TongSearch-QR trains 1.5B and 7B models with GRPO and a frozen-embedding reward to rewrite queries, reaching 27.9 nDCG@10 on BRIGHT with BM25, above GPT-4o's 26.5.
-
Adaptive Graph Pruning for Multi-Agent Communication
AGP trains a graph neural network to jointly decide which agents to keep and how strongly they should communicate, and reports state-of-the-art average accuracy across six LLM benchmarks with large token savings.
-
OSC: Cognitive Orchestration through Dynamic Knowledge Alignment in Multi-Agent LLM Collaboration
OSC uses learned Collaborator Knowledge Models and RL-trained communication policies to make LLM agents communicate adaptively, claiming gains on AlpacaEval 2.0 and MT-Bench.
-
OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critique
A 2.5M-example code reasoning dataset with critique traces enables Qwen2.5-based models to surpass prior open-weight distilled models on LiveCodeBench via test-time self-critique selection.
-
ReasonBridge: Efficient Reasoning Transfer from Closed to Open-Source Language Models
ReasonBridge distills 1,000 curated reasoning traces from a closed model into open-source models via hierarchical adapters and guided inference, claiming up to 23% benchmark gains.
Discussion (0). Sign in to comment.