REVIEW 7 cited by
Archon: An Architecture Search Framework for Inference-Time Techniques
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Inference-time techniques, such as repeated sampling or iterative revisions, are emerging as powerful ways to enhance large-language models (LLMs) at test time. However, best practices for developing systems that combine these techniques remain underdeveloped due to our limited understanding of the utility of each technique across models and tasks, the interactions between them, and the massive search space for combining them. To address these challenges, we introduce Archon, a modular and automated framework for optimizing the process of selecting and combining inference-time techniques and LLMs. Given a compute budget and a set of available LLMs, Archon explores a large design space to discover optimized configurations tailored to target benchmarks. It can design custom or general-purpose architectures that advance the Pareto frontier of accuracy vs. maximum token budget compared to top-performing baselines. Across instruction-following, reasoning, and coding tasks, we show that Archon can leverage additional inference compute budget to design systems that outperform frontier models such as OpenAI's o1, GPT-4o, and Claude 3.5 Sonnet by an average of 15.1%.
Forward citations
Cited by 7 Pith papers
-
AoA: Theorem Proving Agent over Abstract Syntax Tree of Redesigned Language
AoA proves theorems by editing a JSON-AST proof tree for the new Minilang language, reporting 2.9–6.9x fewer tokens and 2.3–4.7x lower API cost than Amazon's Isabelle agent with equal or better pass rates.
-
Training Language Models to Cooperate with Inference-Time Controllers
Multi-controller post-training with turn-level GRPO over reusable reasoning modules improves LLM transfer to held-out and shifted inference-time controllers versus single-controller RL.
-
A Workflow-Aware Serving Layer for Agentic Applications
A workflow-aware serving layer compiles per-node model-verifier-backend plans with an ILP and adapts only uncommitted work via pre-solved pressure rungs and residual re-solves.
-
Automated Discovery Has No Universally Superior Harness
No fixed discovery harness is reliably superior across 12 model–problem pairs, OpenEvolve-style recipes underperform simpler alternatives, and online pruning of weak partial runs improves budget-matched performance.
-
Helpful Agent Meets Deceptive Judge: Understanding Vulnerabilities in Agentic Workflows
LLM agents frequently switch correct answers after one round of misleading feedback, and the new WAFER-QA benchmark measures this with web-backed critiques.
-
How Much Backtracking is Enough? Exploring the Interplay of SFT and RL in Enhancing LLM Reasoning
On three controlled tasks, the optimal number of backtracks in SFT warm-up data grows with task difficulty, while trajectory correctness has little effect on final RL performance.
-
Introspection of Thought Helps AI Agents
INoT wraps prompts in XML-defined pseudo-code so an LLM simulates two debating agents internally, reporting better scores and lower tokens than seven baselines.
Discussion (0). Continue with ORCID to comment.