Pith. sign in

REVIEW 11 cited by

AutoMix: Automatically Mixing Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.12963 v5 pith:B2RBQZSD submitted 2023-10-19 cs.CL cs.AI

Pranjal Aggarwal , Aman Madaan , Ankit Anand , Srividya Pranavi Potharaju , Swaroop Mishra , Pei Zhou , Aditya Gupta , Dheeraj Rajagopal
show 5 more authors
Karthik Kappaganthu Yiming Yang Shyam Upadhyay Manaal Faruqui Mausam
This is my paper · ORCID
classification cs.CLcs.AI
keywords automixlanguagemodelschallengingcomputationalcosteffectivelyfive
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) are now available from cloud API providers in various sizes and configurations. While this diversity offers a broad spectrum of choices, effectively leveraging the options to optimize computational cost and performance remains challenging. In this work, we present Automix, an approach that strategically routes queries to larger LMs, based on the approximate correctness of outputs from a smaller LM. Central to Automix are two key technical contributions. First, it has a few-shot self-verification mechanism, which estimates the reliability of its own outputs without requiring extensive training. Second, given that self-verification can be noisy, it employs a POMDP based router that can effectively select an appropriately sized model, based on answer confidence. Experiments across five language models and five challenging datasets show that Automix consistently surpasses strong baselines, reducing computational cost by over 50% for comparable performance.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Workflow-Aware Serving Layer for Agentic Applications

    cs.DC 2026-07 conditional novelty 6.5 of 10

    A workflow-aware serving layer compiles per-node model-verifier-backend plans with an ILP and adapts only uncommitted work via pre-solved pressure rungs and residual re-solves.

  2. CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents

    cs.AI 2026-07 unverdicted novelty 6.0 of 10

    Budget-calibrated recovery routing with conformal risk control lets coding agents match always-escalate solve rates at about 35% of the cost.

  3. R2-Router: A New Paradigm for LLM Routing with Reasoning

    cs.CL 2026-02 conditional novelty 6.0 of 10

    R2-Router jointly selects the LLM and an output-token budget, modeling each model as a quality-cost curve rather than a fixed point, and reports 4-5x cost savings on its new R2-Bench.

  4. Orchestration for Domain-specific Edge-Cloud Language Models

    cs.DB 2025-07 conditional novelty 6.0 of 10

    ECO-LLM jointly selects query processing, retrieval, and model components per query, cutting cost by 60% and latency up to 6x versus model routing in edge-cloud tests.

  5. IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory

    cs.AI 2025-06 conditional novelty 6.0 of 10

    An IRT-based router that models each LLM's latent ability and each query's difficulty outperforms RouterBench on cost-performance reward across ID and OOD benchmarks.

  6. How Often Should a Recommender Call an LLM? Value-Weighted Routing, Monitoring, and Seasonal Robustness

    cs.AI 2026-07 conditional novelty 5.0 of 10

    Value-weighted LLM routing matches difficulty-only recall while raising precision, exposes within-category calibration collapse, and an elastic value-scaled budget absorbs a synthetic Black Friday surge.

  7. vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models

    cs.NI 2026-02 conditional novelty 5.0 of 10

    vLLM Semantic Router routes LLM requests by composing thirteen signal types into Boolean decision policies, with safety, caching, and model-selection plugin chains.

  8. Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Models

    cs.AI 2025-06 conditional novelty 5.0 of 10

    Conformal Arbitrage calibrates a score-gap threshold with conformal risk control so that a primary model can act when confident and defer to a guardian otherwise, with the expected guardrail loss bounded by a user-cho...

  9. Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes

    cs.CL 2026-08 conditional novelty 4.0 of 10

    Statistical classifiers built on LLM activation norms and coordinates match or beat trained MLP heads on coarse intent routing and resist camouflage better, while MLPs win on fine-grained subfield distinctions.

  10. Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques

    cs.LG 2025-06 unverdicted novelty 4.0 of 10

    A survey of LLM routing and hierarchical inference techniques that proposes an unvalidated unified evaluation metric called the Inference Efficiency Score.

  11. ORI: O Routing Intelligence

    cs.CL 2025-02 reject novelty 3.0 of 10

    ORI routes queries by embedding cluster to the best model for the cluster's dominant benchmark, reporting modest gains that are not supported by its own routing rule or evaluation protocol.

Pith tools