Pith. sign in

REVIEW 5 major objections 5 minor 79 references

HASHIRU: Hierarchical Agent System for Hybrid Intelligent Resource Utilization

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a multi-agent framework whose CEO agent dynamically hires, fires, and routes work to specialized LLM employees—preferring cheap local models—outperforms a single strong model on math reasoning benchmarks, with the…

desk verdict A clear integration of known agent ideas whose headline accuracy gains are confounded by tools and stronger employee models; the architecture is worth a serious look, but the current empirical claims outrun the evidence. read the letter →

arxiv 2506.04255 v1 pith:S36R3IR2 submitted 2025-06-01 cs.MA

Kunal Pai , Parth Shah , Harshil Patel This is my paper
classification cs.MA
keywords multi-agentsystemhierarchicalcontroldynamicagentlifecyclehybridintelligencelocalLLMautonomoustoolcreationresource-awaremanagementreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a multi-agent system can beat a strong single LLM by having a central 'CEO' agent decide, at runtime, which specialized worker agents to hire, fire, and invoke, choosing among cheap local models and costlier external APIs under a memory and money budget. The result that matters: on grade-school math (GSM8K) HASHIRU scores 96% versus 61% for the Gemini 2.0 Flash baseline, on JEEBench 80% versus 68.3%, and on SVAMP 92% versus 84%. If these gains come from the management mechanism rather than from the extra tools and larger models it is allowed to call, the framework offers a practical recipe for getting more reasoning capability per dollar from existing LLMs.

What carries the argument

The load-bearing mechanism is the CEO-Employee hierarchy governed by an economic model. The CEO (Gemini 2.0 Flash) decomposes the task, decides which specialized employee agents to hire or fire, and tracks two resources: dollar cost (API fees plus 'starting bonus' and 'salary' costs for local models) and memory (combined VRAM footprint as a percentage of the local GPU budget). Hiring/firing costs make agent turnover expensive, forcing the CEO to keep stable teams; hybrid intelligence means small local models are the default, with larger API models (Gemini 2.5 Flash, DeepSeek-R1) invoked only when task difficulty or budget allows. An autonomous tool-creation loop lets the CEO commission new API endpoints, and a memory store retrieves past failures to guide future decisions.

What would settle it

Run the Gemini 2.0 Flash baseline with the same tool set (web search, Python sandbox, access to Gemini 2.5 Flash and DeepSeek-R1) and the same budget prompts, and measure GSM8K/JEEBench/SVAMP accuracy. If the gap narrows to near zero, the dynamic hiring/firing architecture, not the resources themselves, is doing the work; if the gap persists, the management mechanism is responsible.

Watch

Extended reading notes

Core claim

HASHIRU's central claim is that dynamic, resource-aware hierarchical management itself improves LLM performance: a CEO agent that decomposes tasks, hires specialized employees on demand, charges hiring and invocation fees to discourage churn, and prefers small local models (3B–7B via Ollama) while escalating to external APIs only when justified yields higher accuracy than a single Gemini 2.0 Flash agent running alone, at modest resource cost. The architecture treats agent lifecycle as an economic optimization problem and memory plus autonomous tool creation as mechanisms for the team to improve over time.

Load-bearing premise

The comparisons assume the correct counterfactual is Gemini 2.0 Flash without tools, so that any accuracy gain is attributed to HASHIRU's dynamic management rather than to the web search, code execution, or larger and stronger models HASHIRU is permitted to invoke.

Editorial extensions

If this is right

  • If the architecture is the cause of the gains, any single capable LLM can be upgraded by wrapping it in a HASHIRU-style CEO that hires specialist workers on demand.
  • Local-first model routing becomes a practical default: small 3B–7B models handle the bulk of tokens, and expensive API calls are reserved for the few sub-tasks that need them.
  • The hiring/invocation fee structure provides a concrete mechanism to cap both memory footprint and dollar spend in agent deployments.
  • The 100% JailbreakBench result implies hierarchical delegation need not weaken the CEO model's safety guardrails, provided the CEO retains final control.
  • Autonomous tool creation and memory retrieval give the system a path to self-extension: it can build the tool it lacks and avoid repeating past failures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The big GSM8K gap (96 vs 61) is far larger than the management overhead explanation would predict; the more likely drivers are the tool access and the stronger models HASHIRU invokes, meaning the paper's headline accuracy claim probably overstates the contribution of hiring/firing. A clean ablation would separate the two.
  • The real, defensible contribution may be resource governance: the economic model turns 'use a cheaper model when possible' into an enforceable budget, which is what would make local-first systems work on laptops and edge devices.
  • The memory function is tested only anecdotally; it invites a quantitative followup measuring how retrieval of past errors changes accuracy on repeated benchmark families.
  • One testable extension: calibrate the hiring and invocation fees against measured accuracy gains per dollar, turning the economic model into a data-driven policy rather than a heuristic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. HASHIRU is a hierarchical multi-agent framework in which a single CEO agent (Gemini 2.0 Flash) dynamically hires and fires specialized employee agents, chooses between local and external LLMs, tracks memory and cost budgets, maintains a semantic memory, and can autonomously create new API tools. The paper evaluates the system on academic paper review, a JailbreakBench subset, and several reasoning and knowledge benchmarks, and reports four case studies illustrating self-generated cost models, tool integration, budget management, and memory-based error correction. The central claim is that HASHIRU's dynamic lifecycle management and hybrid resource selection improve accuracy over a strong single model while using resources efficiently.

Significance. If the accuracy and efficiency claims were fully supported, HASHIRU would be a useful practical contribution: it targets a plausible cost-capability frontier by combining small local models, larger external models, and dynamic team composition. The authors ship source code, benchmarks, and a demo, which are valuable for reproducibility and for downstream comparisons. However, the current evaluation does not isolate the architecture's contribution: the baseline is a bare Gemini 2.0 Flash model, while HASHIRU is allowed tools, web search, and stronger external models. Several reported improvements are not statistically significant, and the resource-efficiency claims are qualitative rather than measured. The core idea is defensible, but the experimental evidence in its present form supports only a much weaker claim.

major comments (5)
  1. [Section V-B and Table I] The central comparison does not isolate the architecture's contribution. HASHIRU is described in Section III-D as able to invoke external tools, web search, and stronger employee models (Table I lists Gemini 2.5 Flash for MMLU Law/Psychology and DeepSeek-R1 7B for Humanity's Last Exam), while the baseline in Section V-B is Gemini 2.0 Flash alone with no tools. The headline gains (GSM8K 96% vs. 61%, JEEBench 80% vs. 68.3%, SVAMP 92% vs. 84%, MMLU Math 91.8% vs. 87.2%) are therefore attributable to tool access or model substitution rather than to dynamic hiring/firing or the economic model. Since Section VII explicitly lists ablation as future work, no experiment isolates the hierarchical management component; the authors should add either a tool-augmented single-agent baseline (Gemini 2.0 Flash with the same tools and retrieval) or an ablation with a static, pre-selected team.
  2. [Section VI and Table I] The statistical support for the headline claim is weak. Only four of the ten reported comparisons reach p < 0.05, and those are exactly the comparisons in which HASHIRU had access to tools or additional models. ARC (96.5 vs. 95), Humanity's Last Exam (5 vs. 2.5), StrategyQA (85 vs. 82), MMLU Law (58.4 vs. 61.6), and MMLU Psychology (78.7 vs. 78.3) are all p > 0.05, with MMLU Law showing a numeric deficit. The subsets are small (40 to 127 items), yet no confidence intervals, effect sizes, or per-question test details are reported. The abstract's phrasing 'outperforming Gemini 2.0 Flash on GSM8K...' omits the non-significant and negative results and overstates what the evidence supports.
  3. [Section III-C, Section III-E and Table I] The resource-efficiency claims are not actually measured. The 'Resource Use' column of Table I consists of qualitative labels (Negligible, Low, Moderate to High), and the only quantitative resource metric is wall-clock time. There are no measurements of API cost, VRAM footprint, or the economic model's parameters (starting bonus, invocation salary, expense cost, VRAM budget, and retrieval similarity threshold are all free parameters). Since contribution (4) and part of contribution (1) are specifically about economic-model-driven resource allocation, the paper needs at least a cost accounting table and a sensitivity analysis for the economic parameters; otherwise the claim that the economic model 'promotes efficient resource allocation' is untested.
  4. [Section V-A and Section V-B] The paper-review and safety evaluations do not provide the comparisons needed to support their claims. For the ICLR 2023 paper review task, there is no baseline at all, and the prompt in Section V-A explicitly says 'Create THREE agents with relevant personalities,' which is a fixed manual specification rather than evidence of dynamic hiring. For JailbreakBench, the authors state that they test only whether the CEO handles prompts safely 'without delegation'; this does not test whether delegation creates a vulnerability, nor does it compare against a baseline, so a '100% success rate' is at best evidence of non-degradation of the CEO model, not evidence of the system's safety under delegation.
  5. [Section IV and Section VII] The case studies are presented as demonstrations of self-improvement, but they are self-referential in a way that weakens the evidence: the system generates its own cost model and tools, and those artifacts are then cited as evidence of the system's capability. The paper itself acknowledges in Section VI that the memory function was 'not quantitatively benchmarked here,' and Section VII lists calibration, ablation, and cost-benefit analysis as future work. For a journal submission, at least one quantitative evaluation of the memory or tool-creation component should be provided.
minor comments (5)
  1. [Table I] The table uses 'SV AMP' while the text and reference list use 'SVAMP'; please standardize the spelling.
  2. [Section VI] The discussion of MMLU Law says there was a 'lack of statistically significant HASHIRU outperformance' even though HASHIRU scored lower than the baseline; this should be phrased as a numeric deficit.
  3. [Section V-A] The benchmark subset sizes are listed in the text (40, 100, 120, 112, 110, 127), but the table does not report them; please add sample sizes to Table I so the p-values can be interpreted.
  4. [Section V-C] Wall-clock time on a MacBook M1 2020 is hardware-specific and unlikely to reproduce; please report the model API versions, exact reasoning prompts, number of runs, and any randomness in the CEO's hiring/firing decisions.
  5. [Section I and Section V-A] The introduction motivates autonomous tool creation, but the only quantitative results that involve tools are accuracy numbers on reasoning benchmarks; the tool-creation examples in Section IV and Appendix B are screenshots without error rates or success metrics.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HASHIRU's accuracy claims rest on external benchmarks, and the main experimental weaknesses are confounded baselines and stipulated task prompts, not definitional or self-citation loops.

full rationale

The paper's central quantitative claims (GSM8K, JEEBench, SVAMP, MMLU, ARC, StrategyQA, Humanity's Last Exam) are measured against external datasets with fixed ground truth, so no benchmark number is defined in terms of HASHIRU's own outputs or fitted parameters. The CEO model (Gemini 2.0 Flash) is the same model as the baseline, but the comparison is confounded because HASHIRU is allowed tools, web search, and employee models while the baseline is not; Section V-B states the baseline choice is meant 'to isolate the impact of our dynamic management and hybrid intelligence features,' yet the isolation is imperfect. Similarly, the paper-review prompt in Section V-A explicitly instructs 'Create THREE agents with relevant personalities, expertise, and review styles,' so the behavioral evidence for autonomous dynamic hiring in that task is partly stipulated by the evaluation input rather than autonomously chosen. These are experimental-validity limitations, which the paper partially acknowledges in Section VII by listing 'system explainability through ablation' as future work. The case studies in Section IV show HASHIRU generating its own cost model, tools, and budget rules; these are demonstrations of autonomy and are not used as fitted inputs to fabricate the external benchmark results. The only self-citations (refs. [46] and [60]) are related-work references in the introduction and future-work discussion and are not load-bearing for the architecture's claims. No equation, fitted parameter, or uniqueness theorem is invoked such that a claimed prediction reduces by construction to the paper's own inputs.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The system's central claims rest on several design choices that are not measured or ablated: the CEO's prompt-driven decisions, the benefit of local-first routing, and the economic model's stability effects. The evaluation also assumes small benchmark subsets are representative. No free parameters are fitted to data, but several important constants (hiring cost, invocation cost, memory threshold, VRAM budget) are unspecified or hand-picked.

free parameters (4)
  • Starting bonus (hiring cost) = None stated
    One-time cost for instantiating local models; no numeric values given, yet the economic model is a central contribution. Section III-C says it can be 'quantitatively scaled' but never specifies scales.
  • Invocation cost (salary) = None stated
    Recurring cost per use of local model; no values or calibration procedure given. Section III-C.
  • Memory retrieval similarity threshold = Not specified
    Memory retrieval uses cosine similarity with a 'predefined threshold' but the threshold is not reported; system behavior depends on it. Section III-G.
  • VRAM budget = 16 GiB example only
    Resource monitoring uses a percentage of total local-model GPU budget; the example is 16 GiB, but no actual budget used in the reported experiments is stated. Section III-E.
assumptions (6)
  • domain assumption Gemini 2.0 Flash, prompted as CEO, can reliably decompose tasks, judge employee performance, and make correct hiring/firing/tool decisions from natural-language instructions.
    This is the core mechanism of the architecture; no ablation verifies decision quality. Section III-B.
  • domain assumption Small local models (3B-7B via Ollama) are cost-effective and capable enough for many subtasks, making the local-first strategy genuinely efficient.
    The cost model is self-generated in a case study but not validated against manual cost accounting. Section III-D, Section IV-A.
  • ad hoc to paper The economic model (starting bonus, salary, expense cost) improves team stability and resource allocation.
    No experiments compare with and without economic frictions, and no parameter values are given. Section III-C.
  • domain assumption The small benchmark subsets (40-127 questions) are representative of full benchmarks.
    Used in all evaluations; no subset construction details or power analysis. Section V-A.
  • ad hoc to paper The performance gap over the baseline is due to the architecture rather than to the additional tools and external models HASHIRU can call.
    This is exactly what the baseline comparison fails to control, so the central interpretation assumes it. Section V-B.
  • standard math The t-test is valid for comparing accuracies on these subsets.
    The paper invokes Student's t-test; this assumes independence across items and approximately normal sampling distributions, which are reasonable but unexamined at n=40-127. Section V-B.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HASHIRU: Hierarchical Agent System for Hybrid Intelligent Resource Utilization." pith.science (2026). https://pith.science/paper/S36R3IR2

@misc{pith2026250604255,
  author       = {Pith},
  title        = {Pith review of: HASHIRU: Hierarchical Agent System for Hybrid Intelligent Resource Utilization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S36R3IR2}},
  note         = {Machine review of arXiv:2506.04255}
}
read the original abstract

Rapid Large Language Model (LLM) advancements are fueling autonomous Multi-Agent System (MAS) development. However, current frameworks often lack flexibility, resource awareness, model diversity, and autonomous tool creation. This paper introduces HASHIRU (Hierarchical Agent System for Hybrid Intelligent Resource Utilization), a novel MAS framework enhancing flexibility, resource efficiency, and adaptability. HASHIRU features a "CEO" agent dynamically managing specialized "employee" agents, instantiated based on task needs and resource constraints (cost, memory). Its hybrid intelligence prioritizes smaller, local LLMs (via Ollama) while flexibly using external APIs and larger models when necessary. An economic model with hiring/firing costs promotes team stability and efficient resource allocation. The system also includes autonomous API tool creation and a memory function. Evaluations on tasks like academic paper review (58% success), safety assessments (100% on a JailbreakBench subset), and complex reasoning (outperforming Gemini 2.0 Flash on GSM8K: 96% vs. 61%; JEEBench: 80% vs. 68.3%; SVAMP: 92% vs. 84%) demonstrate HASHIRU's capabilities. Case studies illustrate its self-improvement via autonomous cost model generation, tool integration, and budget management. HASHIRU offers a promising approach for more robust, efficient, and adaptable MAS through dynamic hierarchical control, resource-aware hybrid intelligence, and autonomous functional extension. Source code and benchmarks are available at https://github.com/HASHIRU-AI/HASHIRU and https://github.com/HASHIRU-AI/HASHIRUBench respectively, and a live demo is available at https://hashiruagentx-hashiruai.hf.space upon request.

Figures

Figures reproduced from arXiv: 2506.04255 by the authors.

Figure 1
Figure 1. High-level architecture of the HASHIRU system, illustrating the CEO [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. HASHIRU’s autonomous budget management system, ensuring [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. HASHIRU enforcing a stored dietary preference in dialogue using its [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: HASHIRU updating and deleting its memory based on user input. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 7
Figure 7. Figure 7: Continuation of the autonomous tool creation example from Figure [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

79 extracted references · 52 canonical work pages

  1. [1]

    The Claude 3 model family: Opus, Sonnet, Haiku

    Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku. Model Card, March 2024. Accessed: 2025-05-01

    work page 2024

  2. [2]

    Have LLMs advanced enough? a challenging problem solving benchmark for large language models

    Daman Arora, Himanshu Singh, and Mausam. Have LLMs advanced enough? a challenging problem solving benchmark for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 7527–7543, Singapore, December 2023. Association for Computational Linguistics

    work page 2023

  3. [3]

    Self-rag: Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, 2023

    work page 2023

  4. [4]

    Emergent autonomous scientific research capabilities of large language models

    Daniil A Boiko, Robert MacKnight, and Gabe Gomes. Emergent autonomous scientific research capabilities of large language models. arXiv preprint arXiv:2304.05332 , 2023

    arXiv 2023

  5. [5]

    A systematic classification of knowledge, reasoning, and context within the arc dataset

    Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yu- vraj, Rajarshi Das, Andrew McCallum, Maria Chang, Achille Fokoue- Nkoutche, Pavan Kapanipathi, Nicholas Mattei, et al. A systematic classification of knowledge, reasoning, and context within the arc dataset. arXiv preprint arXiv:1806.00358 , 2018

    arXiv 2018

  6. [6]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020

    1901

  7. [7]

    Pappas, Florian Tram `er, Hamed Hassani, and Eric Wong

    Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym An- driushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nico- las Flammarion, George J. Pappas, Florian Tram `er, Hamed Hassani, and Eric Wong. Jailbreakbench: An open robustness benchmark for jail- breaking large language models. In NeurIPS Datasets and Benchmarks Track, 2024

    work page 2024

  8. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

Show all 79 references
  1. [9]

    Clearwater, editor

    Scott H. Clearwater, editor. Market-Based Control: A Paradigm for Distributed Resource Allocation . World Scientific, 1996

    1996

  2. [10]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Hee- woo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. In Advances in Neural Informa- tion Pr...

    2021

  3. [11]

    CrewAI Inc. Crewai. https://www.crewai.com/, 2025. Accessed: 2025-05-01

    2025

  4. [12]

    DeepSeek-R1: Incentivizing Reasoning Capa- bility in LLMs via Reinforcement Learning

    DeepSeek-AI and others. DeepSeek-R1: Incentivizing Reasoning Capa- bility in LLMs via Reinforcement Learning. 2025. arXiv:2501.12948

    2025 arXiv

  5. [13]

    Bert: Pre-training of deep bidirectional transformers for language un- derstanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language un- derstanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human lan...

    2019

  6. [14]

    Multi-agent systems: A survey

    Ali Dorri, Salil S Kanhere, and Raja Jurdak. Multi-agent systems: A survey. Ieee Access, 6:28573–28593, 2018

    2018

  7. [15]

    Agent-organized networks for dynamic team formation

    Matthew E Gaston and Marie DesJardins. Agent-organized networks for dynamic team formation. In Proceedings of the fourth international joint conference on Autonomous agents and multiagent systems , pages 230–237, 2005

    2005

  8. [16]

    Agent-organized networks for multi-agent production and exchange

    Matthew E Gaston and Marie DesJardins. Agent-organized networks for multi-agent production and exchange. In Proceedings of the 20th national conference on Artificial intelligence-Volume 1 , pages 77–82, 2005

    2005

  9. [17]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. 2024. arXiv:2403.05530

    2024 arXiv

  10. [18]

    Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics (TACL) , 2021

    2021

  11. [19]

    Gemini 1.5 flash-8b: Production- ready lightweight model

    Google DeepMind and Google AI. Gemini 1.5 flash-8b: Production- ready lightweight model. https://developers.googleblog.com/en/gem ini-15-flash-8b-is-now-generally-available-for-use/, 2024. Accessed: 2025-05-24

    2024

  12. [20]

    Gemini 1.5 flash: Lightweight multimodal model

    Google DeepMind and Google AI. Gemini 1.5 flash: Lightweight multimodal model. https://cloud.google.com/vertex-ai/generative-a i/docs/models/gemini/1-5-flash, 2024. Accessed: 2025-05-24

    2024

  13. [21]

    Gemini 2.0 flash: Model card, api, and announcement

    Google DeepMind and Google AI. Gemini 2.0 flash: Model card, api, and announcement. https://developers.googleblog.com/en/start-b uilding-with-the-gemini-2-0-flash-family/, 2025. See also: https: //console.cloud.google.com/vertex-ai/publishers/google/model-garde n/gemini-2.0-fl...

    2025

  14. [22]

    Gemini 2.5 flash: Model card, api, and announcement

    Google DeepMind and Google AI. Gemini 2.5 flash: Model card, api, and announcement. https://developers.googleblog.com/en/start-buildin g-with-gemini-25-flash/, 2025. See also: https://console.cloud.google.c om/vertex-ai/publishers/google/model-garden/gemini-2.5-flash-preview -...

    2025

  15. [23]

    Groq: Fast ai inference, 2025

    Groq, Inc. Groq: Fast ai inference, 2025. Accessed: 2025-05-22

    2025

  16. [24]

    Measuring massive multi- task language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multi- task language understanding, 2021

    2021

  17. [25]

    A survey of multi-agent organizational paradigms

    Bryan Horling and Victor Lesser. A survey of multi-agent organizational paradigms. The Knowledge engineering review , 19(4):281–316, 2004

    2004

  18. [26]

    Hugging face: The ai community building the future,

    Hugging Face, Inc. Hugging face: The ai community building the future,

  19. [27]

    Mistral 7B

    Albert Q Jiang, Alexandre Xu, Arthur Mensch Guillaume Lam- ple Nicol `as Lachaux, Franc ¸ois Rozenberg, Timoth´ee Lacroix, Thibaut Lavril, Teven Le Scao Eleonora Gaddipati, Lucile Saulnier Lixin Ortiz, Dieuwke Hiemstra L ´elio Renard Tang, et al. Mistral 7B. 2023

    2023

  20. [28]

    Agentreview: Exploring peer review dynamics with llm agents, 2024

    Yiqiao Jin, Qinlin Zhao, Yiyang Wang, Hao Chen, Kaijie Zhu, Yijia Xiao, and Jindong Wang. Agentreview: Exploring peer review dynamics with llm agents, 2024

    2024

  21. [29]

    Lambda: Gpu cloud and deep learning workstations,

    Lambda Labs. Lambda: Gpu cloud and deep learning workstations,

  22. [30]

    Langgraph: A framework for agentic workflows

    LangChain. Langgraph: A framework for agentic workflows. https: //www.langchain.com/langgraph, 2024. Accessed: May 1, 2025

    2024

  23. [31]

    Accessed: 2025-05-22

    2025

  24. [32]

    Self-evolving agents with reflective and memory- augmented abilities

    Xuechen Liang, Meiling Tao, Yinghui Xia, Tianyu Shi, Jun Wang, and JingSong Yang. Self-evolving agents with reflective and memory- augmented abilities. arXiv preprint arXiv:2409.00872 , 2024

    2024 arXiv

  25. [33]

    Retrieval- augmented generation for knowledge-intensive nlp tasks, 2021

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K ¨uttler, Mike Lewis, Wen tau Yih, Tim Rockt ¨aschel, Sebastian Riedel, and Douwe Kiela. Retrieval- augmented generation for knowledge-intensive nlp tasks, 2021

    2021

  26. [34]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.0...

    2024 arXiv

  27. [35]

    Qa-calibration of language model confidence scores

    Putra Manggala, Atalanti A Mastakouri, Elke Kirschbaum, Shiva Ka- siviswanathan, and Aaditya Ramdas. Qa-calibration of language model confidence scores. In The Thirteenth International Conference on Learning Representations

  28. [36]

    Guest editors’ introduction: Intelligent control in the manufacturing supply chain

    Duncan McFarlane, Vladim ´ır Mar ´ık, and Paul Valckenaers. Guest editors’ introduction: Intelligent control in the manufacturing supply chain. IEEE Intelligent Systems , 20(1):24–26, 2005

    2005

  29. [37]

    Tdc 2023 (llm edition): The trojan detection challenge

    Mantas Mazeika, Andy Zou, Norman Mu, Long Phan, Zifan Wang, Chunru Yu, Adam Khoja, Fengqing Jiang, Aidan O’Gara, Ellie Sakhaee, Zhen Xiang, Arezoo Rajabi, Dan Hendrycks, Radha Poovendran, Bo Li, and David Forsyth. Tdc 2023 (llm edition): The trojan detection challenge. In Neur...

    2023

  30. [38]

    The Llama 3 Herd of Models

    Meta Llama Team. The Llama 3 Herd of Models. 2024. arXiv:2407.21783

    2024 arXiv

  31. [39]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation

    Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. Weblog post, April 2025. Accessed: May 30, 2025

    2025

  32. [40]

    Ollama Team. Ollama. https://ollama.com/, 2023. Accessed: 2025-05- 01

    2023

  33. [41]

    Mistral saba

    Mistral AI. Mistral saba. Weblog post, February 2025. Accessed: May 30, 2025

    2025

  34. [42]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023

    2023

  35. [43]

    Function calling

    OpenAI. Function calling. OpenAI API Documentation, 2023. Ac- cessed: 2025-05-01

    2023

  36. [44]

    Sudden high costs for chatgpt api usage, 2025

    OpenAI Community. Sudden high costs for chatgpt api usage, 2025

    2025

  37. [45]

    Sos: Alarming situation of excessive billing, 2025

    OpenAI Community. Sos: Alarming situation of excessive billing, 2025

    2025

  38. [46]

    CoDocBench: A dataset for code-documentation alignment in software maintenance

    Kunal Pai, Premkumar Devanbu, and Toufique Ahmed. CoDocBench: A dataset for code-documentation alignment in software maintenance. arXiv preprint arXiv:2502.00519 , 2024

    2024 arXiv

  39. [47]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wain- wright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27...

    2022

  40. [48]

    O’Brien, Carrie J

    Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In The 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23) , UIST ’23, page 1...

    2023

  41. [49]

    Talm: Tool augmented language models, 2022

    Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models, 2022

    2022

  42. [50]

    Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191, 2021

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191, 2021

    2021 arXiv

  43. [51]

    Are NLP models really able to solve simple math word problems? In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2021

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2021. Introduces the SV AMP challenge dataset

    2021

  44. [52]

    Chat- dev: Communicative agents for software development

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. Chat- dev: Communicative agents for software development. arXiv preprint arXiv:2307.07924, 2023

    2023 arXiv

  45. [53]

    Humanity’s last exam, 2025

    Long Phan, Alice Gatti, Ziwen Han, et al. Humanity’s last exam, 2025

    2025

  46. [54]

    Qwen2.5 Technical Report

    Qwen Team, An Yang, et al. Qwen2.5 Technical Report. 2024. arXiv:2412.15115

    2024 arXiv

  47. [55]

    Toolllm: Facilitating large language models to master 16000+ real-world APIs, 2023

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Shijie Wang, Zelin Lu, Siyu Xi, Xiao Liu, Yongyan Li, Zihan Wang, Zixuan Liu, Jian- Guang Lou, et al. Toolllm: Facilitating large language models to master 16000+ real-world APIs, 2023. Accessed: May 26, 2025

    2023

  48. [56]

    0.56 to $343.15 in minutes – google gemini api, 2025

    Reddit user. 0.56 to $343.15 in minutes – google gemini api, 2025

    2025

  49. [57]

    Explor- ing the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Explor- ing the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1–67, 2020

    2020

  50. [58]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634– 8652, 2023

    2023

  51. [59]

    Russell and Peter Norvig

    Stuart J. Russell and Peter Norvig. Artificial intelligence: a modern approach. Prentice Hall Press, Upper Saddle River, NJ, USA, 3rd edition, 2010

    2010

  52. [60]

    Calibration and correctness of language models for code

    Claudio Spiess, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam Rabin, Amin Alipour, Susmit Jha, Prem Devanbu, and Toufique Ahmed. Calibration and correctness of language models for code. arXiv preprint arXiv:2402.02047 , 2024

    2024 arXiv

  53. [61]

    Agent-oriented programming

    Yoav Shoham. Agent-oriented programming. Artificial Intelligence , 60(1):51–92, 1993

    1993

  54. [62]

    Hermes 3 technical report

    Ryan Teknium, Jeffrey Quesnelle, and Chen Guang. Hermes 3 technical report. arXiv preprint arXiv:2408.11857 , 2024

    2024 arXiv

  55. [63]

    The probable error of a mean

    Student. The probable error of a mean. Biometrika, pages 1–25, 1908

    1908

  56. [64]

    V oyager: An open-ended embodied agent with large language models, 2023

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An open-ended embodied agent with large language models, 2023

    2023

  57. [65]

    Qwq-32b: Embracing the power of reinforcement learning

    The Qwen Team. Qwq-32b: Embracing the power of reinforcement learning. Weblog post, March 2025. Accessed: May 30, 2025

    2025

  58. [66]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers, 2020

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers, 2020

    2020

  59. [67]

    A survey on large language model based autonomous agents, 2023

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. A survey on large language model based autonomous agents, 2023

    2023

  60. [68]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural infor- mation processing systems , 35:24824–24837, 2022

    2022

  61. [69]

    Decision-making driven by driver intelligence and environment reasoning for high-level autonomous vehi- cles: a survey

    Yuning Wang, Junkai Jiang, Shangyi Li, Ruochen Li, Shaobing Xu, Jianqiang Wang, and Keqiang Li. Decision-making driven by driver intelligence and environment reasoning for high-level autonomous vehi- cles: a survey. IEEE Transactions on Intelligent Transportation Systems, 24(1...

    2023

  62. [70]

    An introduction to multiagent systems

    Michael Wooldridge. An introduction to multiagent systems. John wiley & sons, 2009

    2009

  63. [71]

    Benchmarking complex instruction- following with multiple constraints composition, 2024

    Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Yiming Liu, Jie Tang, Hongning Wang, and Minlie Huang. Benchmarking complex instruction- following with multiple constraints composition, 2024

    2024

  64. [72]

    The rise and potential of large language model based agents: A survey, 2023

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Linyi Yang, Ting Ruan, Yongquan Yang, Peng Li, Yitao Chang, and Yanlin Wang. The rise and potential of large lan...

    2023

  65. [73]

    Awadallah, Ryen W

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed H. Awadallah, Ryen W. White, Doug Burger, and Chi Wang. AutoGen: Enabling next-gen LLM applications via multi-agent conversation. arXiv preprint arX...

    2023 arXiv

  66. [74]

    Tenenbaum, Tianmin Shu, and Chuang Gan

    Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models, 2023

    2023

  67. [75]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Repre- sentations (ICLR), 2023. arXiv:2210.03629

    2023 arXiv

  68. [76]

    Agents: An open-source framework for large language model based autonomous agents, 2023

    Wangchunshu Zhou, Jianshu Chen, Jialong Wu, Yiheng Xu, Kexin Wang, Jintian Zhang, Yuan Gao, Zhiyong Wu, Kevin Tian, Yubo Feng, Linyi Yang, Bokai Quan, Cong Yu, Yuhang Wang, Shishen Lan, Yan Wang, Hong-Cheng Guo, Chaoyu Chen, Tianxiang Sun, Jin Xiong, Yi Lu, Peng Li, Lichao Sun...

    2023

  69. [77]

    Is llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks

    Ruiyang Zhou, Lu Chen, and Kai Yu. Is llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks. In Proceedings of the 2024 Joint International Conference on Computa- tional Linguistics, Language Resources and Evaluation (LREC-COLING 2024), ...

    2024

  70. [78]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023. APPENDIX In this section, we provide examples of how HASHIRU operates in practice, demonstrating its dynamic management, hybrid intellige...

    2023

  71. [79]

    Fine- tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine- tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.