REVIEW 7 cited by
Learning How Hard to Think: Input-Adaptive Allocation of LM Computation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Computationally intensive decoding procedures--including search, reranking, and self-critique--can improve the quality of language model (LM) outputs in problems spanning code generation, numerical reasoning, and dialog. Existing work typically applies the same decoding procedure for every input to an LM. But not all inputs require the same amount of computation to process. Can we allocate decoding computation adaptively, using more resources to answer questions whose answers will be harder to compute? We present an approach that predicts the distribution of rewards given an input and computation budget, then allocates additional computation to inputs for which it is predicted to be most useful. We apply this approach in two decoding procedures: first, an adaptive best-of-k procedure that dynamically selects the number of samples to generate as input to a reranker; second, a routing procedure that dynamically responds to a query using a decoding procedure that is expensive but accurate, or one that is cheaper but less capable. Across a suite of programming, mathematics, and dialog tasks, we show that accurate computation-allocation procedures can be learned, and reduce computation by up to 50% at no cost to response quality, or improve quality by up to 10% at a fixed computational budget.
Forward citations
Cited by 7 Pith papers
-
Latency and Token-Aware Test-Time Compute
A learned per-query router selects both the inference-scaling method and its compute budget to balance accuracy, token use, and latency, outperforming static strategies on math reasoning.
-
BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokens
A control-token insertion and two-stage training method that lets LLMs adhere to user-specified reasoning token budgets while preserving math accuracy.
-
Structured Pruning for Diverse Best-of-N Reasoning Optimization
SPRINT learns to select which attention heads to prune per question, improving Pass@N over random head selection and multinomial sampling on MATH500 and GSM8K.
-
EquivPruner: Boosting Efficiency and Quality in LLM-Based Search via Action Pruning
EquivPruner detects mathematically equivalent reasoning steps during LLM tree search and keeps only one per group, cutting token use by up to half on GSM8K and MATH-500 without hurting accuracy.
-
Emergent Response Planning in LLMs
Hidden representations of LLM prompts encode global attributes of the upcoming response, and simple probes can predict length, content choices, and answer confidence before generation begins.
-
From Long to Short: LLMs Excel at Trimming Own Reasoning Chains
EDIT searches across step-count prompts to find the shortest reasoning path the model consistently answers correctly.
-
Less is More Tokens: Efficient Math Reasoning via Difficulty-Aware Chain-of-Thought Distillation
Difficulty-aware compression of CoT traces plus SFT and DPO lets LLMs shorten reasoning on easy math problems, cutting tokens by up to 30% with mixed accuracy effects.
Discussion (0). Continue with ORCID to comment.