Pith. sign in

REVIEW 4 major objections 4 minor 55 references

Static leaderboards can crown the worse system to deploy when compute is counted.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 23:06 UTC pith:AMZMQTIX

load-bearing objection The headline reversal is not reproducible from the paper's own equations, and the comparison violates its own match-specific definition; the cost-aware protocol idea is still worth a look. the 4 major comments →

arxiv 2607.24063 v2 pith:AMZMQTIX submitted 2026-07-27 cs.AI

The Cost of Knowing: A Resource-Aware Protocol for Benchmarking Hallucination Beyond Static Leaderboards

Keyu Li , Jin Gao , Dequan Wang This is my paper
classification cs.AI
keywords hallucination evaluationresource-aware benchmarkscost of factualityQ-Scoremulti-agent competitionfactuality-cost trade-offstatic leaderboards
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Static factuality leaderboards treat compute as free, so a system can buy a higher score by spending more tokens, time, or review cycles. This paper argues that this makes leaderboards misleading, and demonstrates a ranking reversal on summarization: a brute-force Best-of-4 agent posts the higher raw factuality score but is the worse system once operational cost is counted, at roughly four times the tokens and latency. To make the trade-off visible, the paper introduces MAS-HQ, a protocol that wraps any factuality detector, pits agents against each other under a shared budget, and scores them with a Q-Score equal to factuality minus normalized cost. Across summarization and QA, competition with scored cost shifts agents toward more frugal policies, and the cost-adjusted axis stays discriminative even for frontier models whose raw scores are bunched near the ceiling. A sympathetic reader would take away that cost-aware evaluation can change deployment recommendations and reveal over-optimization that static benchmarks hide.

Core claim

The paper's central claim is that the system topping a static factuality leaderboard can be the worse one to deploy, because raw factuality ignores the compute used to buy it. It supports this with a direct reversal: on a summarization benchmark, a brute-force Best-of-4 agent reaches a higher H-Score (0.9169) than a Q-Agent (0.9103), yet loses on the cost-adjusted Q-Score (0.5169 vs 0.5217) while consuming roughly four times the tokens and latency. The Q-Score subtracts a match-normalized resource penalty from factuality, so an accuracy gain bought with disproportionate compute is scored down. The paper reports that the same pattern recurs across several models and persists for frontier syst

What carries the argument

The load-bearing object is the Q-Score, defined per match as 1/N Σ_i (α·H-Score_i − β·P_i), where H-Score is the plugged-in factuality metric from any detector, P_i is a cost penalty aggregating tokens, API calls, review iterations, and wall-clock time—max-normalized over the agents in the same match—and α, β are deployment-specified weights for accuracy versus cost. Because the penalty is relative to co-present competitors, the Q-Score is a match-specific payoff rather than a standalone model score. The competitive setup, with two agents processing the same passage bank in opposite orders and an optional telemetry knob that reveals opponent state when a costly review occurs, creates the asy

Load-bearing premise

The reversal assumes that a Q-Score computed against one opponent can be directly compared to a Q-Score computed against another opponent, even though the penalty term is max-normalized within each match.

What would settle it

Run the Best-of-4 baseline and the Q-Agent in the same match, so both are normalized against the same co-present competitor; if Best-of-4's Q-Score still exceeds the Q-Agent's, the claimed reversal is false. Alternatively, run a round-robin tournament of all systems with the same cost weight and check whether Best-of-4's aggregate cost-adjusted rank is lower than the Q-Agent's.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Deployment choices made from raw factuality rankings can be suboptimal; a cost-honest score flips the ranking between a brute-force sampler and a more frugal agent.
  • If leaderboards adopted cost-adjusted scoring, brute-force accuracy-buying strategies would be penalized, shifting optimization pressure toward efficiency.
  • The cost-adjusted axis remains discriminative even when raw factuality saturates near the ceiling, so it can separate frontier systems that static scores cannot.
  • The protocol is metric-agnostic: any factuality detector can be substituted, and the framework transfers to exact-match QA and to N-player matches.
  • Scoring cost changes observed behavior: under a shared budget, agents reduce review cycles and token use while retaining most of the factual gain.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: If cost-aware scores became standard, benchmark incentives would flip from 'spend more to gain a fraction of a point' to 'spend less for the same point,' which could change how providers optimize models.
  • Editorial inference: The paper's own definition implies Q-Scores from different matches are not directly comparable; turning MAS-HQ into a global ranking would require tournament-style aggregation, a step the paper leaves to future work.
  • Editorial inference: The same competition mechanism could be applied to other scarce dimensions—monetary API cost, energy, or information leakage—and to other quality axes beyond factuality.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MAS-HQ, a resource-aware evaluation protocol that scores systems by a Q-Score defined as factuality (H-Score) minus a normalized operational-cost penalty under a competitive match, and introduces a modular Q-Agent reference architecture with an optional telemetry knob. The central claim is a ranking reversal: a brute-force Best-of-4 agent attains the higher H-Score but, once cost is penalized, receives a lower Q-Score than a Q-Agent, implying that static leaderboards can recommend the wrong deployment pick. Supporting experiments claim that competition elicits resource-efficient policies, that the effect persists for frontier models, and that it is stable across 100 trials.

Significance. If the reversal were established, the paper would make a timely and practically relevant point: static factuality leaderboards ignore the compute cost of accuracy, and cost-aware evaluation can change deployment choices. The paper also includes useful components: a metric-agnostic protocol, a blind human validation of the factuality detector (Pearson r = 0.96), a ReAct-based architecture-agnostic check, and a 100-trial stability analysis. However, the main reversal is not reproducible from the formulas and tables as printed, and the comparison violates the paper's own match-specific definition of Q-Score. The contribution is therefore not yet supported by the evidence presented.

major comments (4)
  1. [MAS-HQ Mechanics, Eq. (1); Tables 2, 4, 13] The Q-Score arithmetic is internally inconsistent. With α=1 and β=0.01, and P_i max-normalized over agents in the same match, a penalty of about 0.39 per unit (as implied by Table 2: Q = 0.9103−0.5217 = 0.3886) is impossible if P_i ∈ [0,1]. The same GPT-4o-mini Q-Agent A run appears in Table 2 with Q-Score 0.5217 and in Table 13 with Q-Score 0.8715, despite identical H-Score and resource metrics; the latter corresponds to a penalty of 0.03886, a factor of 10 smaller. Table 4 reports β=0 with a nonzero Resource Penalty (0.3899) and Q-Score 0.5267, contradicting the definition Q = Σ(α·H − β·P) at β=0. These discrepancies mean the headline reversal values do not follow from the stated formula and cannot be independently checked.
  2. [MAS-HQ Mechanics; Table 2] The central comparison violates the paper's own definition. The text states that 'the Q-Score is a match-specific payoff and not a standalone model score,' because P_i is max-normalized over the agents in the same match. Table 2 compares a Single-Agent Best-of-4 baseline (which has no co-present competitor and therefore no well-defined opponent normalization) with Q-Agent rows from a separate match. This is not a valid Q-Score comparison under the protocol. To make the claimed reversal, the Best-of-4 system would need to participate in the same match or be scored through the tournament/Elo aggregation that the paper defers to future work.
  3. [Experiments; Table 4 and Figure 3] The β-sensitivity analysis strengthens the concern that the reversal is built into the penalty scale rather than the data. At β=0.02, Table 4 shows Q-Agent B with higher H-Score (0.9096 vs. 0.9043) but lower Q-Score (0.5096 vs. 0.5142), even though the reported Resource Penalty difference (0.0099) times β is tiny. With β=0.02, the accuracy difference should dominate the cost penalty unless the effective normalizing scale is much larger than the stated max-normalized [0,1] range. This suggests the reported Q-Scores are computed with an implicit 10× cost scale, not the stated β=0.01, and the sensitivity sweep does not recover the claimed behavior from the stated formula.
  4. [Formalization as a POSG; Appendix] The POSG formalization is presented as grounding, but the paper explicitly does not solve the game or compute equilibria. The appendix's Bellman equation and PBE conditions are descriptive only. This is not a flaw by itself, but the language in the main text (e.g., 'strategic tension arises directly from this structure') overstates the grounding. Since the empirical claims are the load-bearing evidence, this appendix does not compensate for the arithmetic and comparison problems above.
minor comments (4)
  1. [Abstract and Introduction] The abstract states 'under a reported cost weight whose sensitivity we sweep,' but the reported weight (β=0.01) is inconsistent with the actual penalty scale used in the tables, as noted above. The phrase should be clarified or revised.
  2. [Table 13 caption] Table 13's caption says it shows 'the detailed data in the main text,' but the Q-Score for the first row (A: GPT-4o-mini) differs from Table 2, and no explanation is given. Either the table is mislabeled or one of the two reported Q-Scores is wrong.
  3. [Experimental Setup] The paper claims 'Reproducible analysis' but does not provide code or raw logs. Releasing the implementation and exact prompts would materially improve the reproducibility of the resource-normalization and Q-Score computation.
  4. [Figure 1] The figure's example shows 'Penalty: 0.2' and 'Penalty: 0.1' with Q-Score = H − Penalty, which is inconsistent with the stated formula Q = H − β·P for any β in [0.01, 0.02]. This visual gloss may reflect the same normalization confusion as the tables.

Circularity Check

0 steps flagged

No significant circularity: the reversal is an explicit arithmetic consequence of the stated Q-Score formula and a reported cost weight, not a fitted or self-cited prediction.

full rationale

The paper defines Q-Score = (1/N) Σ (α·H-Score_i − β·P_i) and explicitly labels α and β as deployment-SLA knobs, reporting sensitivity rather than claiming a canonical value. The ranking reversal of Best-of-4 vs. Q-Agent A is therefore presented as a direct arithmetic consequence of that definition plus measured costs, not as a hidden reduction of a prediction to a fitted input. The empirical claims—reduced review effort under competition, ReAct substitution yielding similar behavior, and 100-trial stability—are measurements made under the stated objective and are not assumed by the metric. Self-citations (e.g., Li, Gao, and Wang 2026 and related preprints) appear only as background references; no load-bearing uniqueness theorem or ansatz is imported from prior work. H-Score is externally anchored to HHEM-2.1 and validated by a reported 0.96 human correlation. The paper does contain non-circularity flaws that a correctness review should weigh: Table 4's β=0 row still subtracts a Resource Penalty (0.9166 − 0.3899 = 0.5267), Table 13 reports the same GPT-4o-mini run with Q-Score 0.8715 while Table 2 reports 0.5217, and Table 2 compares Single-Agent Best-of-N rows to Q-Agent rows even though the paper states Q-Score is a match-specific payoff requiring co-present competitors. These are reproducibility and validity problems, but they are not cases where an output equals an input by construction, so they do not raise the circularity score.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 3 invented entities

The paper's headline reversal is an artifact of several hand-chosen components: α,β, the undocumented 10× penalty scale, and the match-relative normalization. The factuality detector and the exclusion rule are additional assumptions the central comparison depends on. Counting these makes clear that the empirical contribution is a specific evaluation configuration rather than a parameter-free discovery.

free parameters (6)
  • α (accuracy weight) = 1
    Chosen deployment-SLA knob. Q-Score is a linear function of α; the ranking reversal depends on α=1 being prioritized equally with β.
  • β (cost weight) = 0.01 (swept in Table 4)
    Main results use β=0.01. At β=0 the reversal disappears; at β=0.02 cost dominates. The headline reversal is contingent on this hand-selected value.
  • T (hallucination review threshold) = 0.85
    Threshold below which the Policy Agent is nudged to review. Swept in Table 10; affects resource usage and Q-Score.
  • R (maximum review cycles) = 3
    Limits review attempts per passage. Swept in Table 11; best Q-Score chosen at R=3.
  • Implicit 10× cost normalization scale = 10 (implied by table values)
    The stated formula says P_i aggregates max-normalized costs, which should yield values near 4 for four components. The reported Q-Scores imply P_i ≈ 38.9 for β=0.01, i.e., each component is scaled to 0–10 rather than 0–1. This scale is never stated.
  • Equal weights for cost components = 1 each
    API calls, tokens, reviews, and runtime are summed with equal weight; no justification or ablation for this weighting.
axioms (5)
  • domain assumption The HHEM-2.1 discriminator's H-Score is a valid factuality ranking metric.
    A blind human eval on 100 summaries gives Pearson 0.96, but the main comparisons run on ~800 passages and no error bars or inter-annotator agreement are reported.
  • ad hoc to paper Factuality should be linearly traded against normalized operational cost with the chosen α,β.
    The Q-Score objective encodes a specific SLA preference. The paper frames the reversal as an empirical result, but it is a consequence of this adopted objective.
  • domain assumption Excluding passages that trigger safety refusals in any evaluated model does not bias the comparison.
    The paper scores only the intersection of passages all models can process. This is a selection rule that could advantage or disadvantage particular models.
  • domain assumption LLM agents' observed choices (continue/review/end) can be interpreted as resource-efficient policies under the Q-Score incentive.
    The agents are prompted to optimize Q-Score, but it is not independently verified that their behavior is causally driven by the cost term rather than by prompt phrasing.
  • standard math The POSG formalization is a faithful description of the implemented game.
    The paper says the POSG is used only as descriptive vocabulary and no equilibria are solved, so it is not load-bearing, but it is stated as a formal grounding.
invented entities (3)
  • Q-Score no independent evidence
    purpose: A metric that subtracts normalized operational cost from factuality, meant to expose the cost-of-knowing trade-off.
    No external validation; it is a newly defined objective whose parameters α,β and normalization scale are chosen by the authors.
  • Vision mechanism (telemetry knob) no independent evidence
    purpose: Conditionally reveals an opponent's state when a costly review action is taken, introducing partial observability.
    A software mechanism within the benchmark; no independent handle outside this paper's experimental setting.
  • Q-Agent reference architecture no independent evidence
    purpose: A modular Policy/Summary/Review/Evaluation agent that implements MAS-HQ as a reproducible baseline.
    An engineered agent design, not a discovered entity; its behavior is only measured within the proposed protocol.

pith-pipeline@v1.3.0-alltime-deepseek · 19259 in / 14748 out tokens · 127566 ms · 2026-07-31T23:06:15.161982+00:00 · methodology

0 comments
read the original abstract

On standard factuality tasks, frontier models now cluster near the top of the scale. The question is therefore shifting from how factual a system is toward how much compute that factuality costs. Static leaderboards score factuality in isolation and treat compute as free, so they cannot tell a genuinely better system apart from one that simply spends more. Consider a ranking reversal. A brute-force Best-of-4 agent posts the higher raw factuality score (H-Score 0.9169 vs 0.9103) and would top a static leaderboard, but once cost is counted it is the worse system, losing on Q-Score (0.5169 vs 0.5217) at roughly four times the tokens and latency, under a reported cost weight whose sensitivity we sweep. So the system that tops a static leaderboard can be the worse one to deploy. To make this trade-off visible, we introduce MAS-HQ (Multi-Agent System Hallucination Quest), a resource-aware evaluation protocol. It wraps any factuality detector and normalizes for cost, and it pits systems against each other rather than scoring them in isolation. The Q-Score measures factuality minus normalized cost under a competitive match. Across summarization and open-domain QA, single-agent baselines drift into resource-heavy over-optimization, while competition elicits more resource-efficient policies. These gains are small but consistent, and stable across 100 trials. The axis stays discriminative for frontier systems (Gemini-2.5-Pro, and GPT-5) whose raw factuality scores are already bunched near the ceiling. MAS-HQ provides a reproducible way to measure how much a factual answer costs.

Figures

Figures reproduced from arXiv: 2607.24063 by Dequan Wang, Jin Gao, Keyu Li.

Figure 1
Figure 1. Figure 1: Same tasks, reversed ranking: higher factuality, worse once cost is counted. Two Q-Agents compete on MAS-HQ; the original passage list is input in different sequences to Q-Agent A and Q-Agent B, which devise strategies to summarize each passage using mechanisms such as the vision mechanism. Hallucination scores and resource-usage penalties are recorded and combined into the Q-Score, and the agent with the … view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the composition of the Q-Agent. Policy Agent receives both the Q-Agent’s own historical information and the opponent’s historical information (via the vision mechanism) to determine the next strategy. The text summary generated by either the Summary Agent or the Review Agent is evaluated by the Evaluation Agent, which computes the hallucination score (H-Score) and resource usage. After all step… view at source ↗
Figure 3
Figure 3. Figure 3: Ablation study of the Review Agent in Q-Agent. Each experiment group presents the average results over Q-Agent A and Q-Agent B. Left: The Review Agent improves the H-Score and reduces hallucination to varying extents across different Q-Agent models. Right: When both Q-Agents are built using GPT-4o-mini, the smaller the penalty coefficient β, the more the review pays off in the final Q-Score—as β → 0 the ra… view at source ↗
Figure 4
Figure 4. Figure 4: Case study on a subset of MAS-HQ. Through vision mechanism, Q-Agent A and Q-Agent B dynamically adjust their strategies based on their own H-Score, resource usage, and comparisons with the opponent, deciding whether to continue generating summaries or perform reviews. In this example the same reversal appears at the match level: Q-Agent B wins on the cost-honest Q-Score through lower resource usage despite… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 34 linked inside Pith

  1. [1]

    Google DeepMind , title =

  2. [2]

    doi:10.57967/hf/3240 , publisher =

    Forrest Bao and Miaoran Li and Rogger Luo and Ofer Mendelevitch , title =. doi:10.57967/hf/3240 , publisher =

    doi:10.57967/hf/3240

  3. [3]

    2013 , publisher=

    Hallucination: Philosophy and psychology , author=. 2013 , publisher=

    2013

  4. [4]

    ACM Transactions on Information Systems , volume=

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions , author=. ACM Transactions on Information Systems , volume=. 2025 , publisher=

    2025

  5. [5]

    arXiv preprint arXiv:2401.11817 , year=

    Hallucination is inevitable: An innate limitation of large language models , author=. arXiv preprint arXiv:2401.11817 , year=

    Pith/arXiv arXiv

  6. [6]

    arXiv preprint arXiv:2104.14839 , year=

    The factual inconsistency problem in abstractive text summarization: A survey , author=. arXiv preprint arXiv:2104.14839 , year=

    Pith/arXiv arXiv

  7. [7]

    ACM computing surveys , volume=

    Survey of hallucination in natural language generation , author=. ACM computing surveys , volume=. 2023 , publisher=

    2023

  8. [8]

    arXiv preprint arXiv:2203.05227 , year=

    Faithfulness in natural language generation: A systematic survey of analysis, evaluation and optimization methods , author=. arXiv preprint arXiv:2203.05227 , year=

    Pith/arXiv arXiv

  9. [9]

    Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages=

    On the dangers of stochastic parrots: Can language models be too big? , author=. Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages=

    2021

  10. [10]

    arXiv preprint arXiv:2307.00360 , year=

    Batgpt: A bidirectional autoregessive talker from generative pre-trained transformer , author=. arXiv preprint arXiv:2307.00360 , year=

    Pith/arXiv arXiv

  11. [11]

    arXiv preprint arXiv:1904.09751 , year=

    The curious case of neural text degeneration , author=. arXiv preprint arXiv:1904.09751 , year=

    Pith/arXiv arXiv 1904

  12. [12]

    URL https://arxiv

    Truthfulqa: Measuring how models mimic human falsehoods, 2022 , author=. URL https://arxiv. org/abs/2109.07958 , year=

    Pith/arXiv arXiv 2022

  13. [13]

    arXiv preprint arXiv:2310.03368 , year=

    Evaluating hallucinations in chinese large language models , author=. arXiv preprint arXiv:2310.03368 , year=

    Pith/arXiv arXiv

  14. [14]

    arXiv preprint arXiv:2401.03205 , year=

    The dawn after the dark: An empirical study on factuality hallucination in large language models , author=. arXiv preprint arXiv:2401.03205 , year=

    Pith/arXiv arXiv

  15. [15]

    URL https://arxiv

    Semdedup: Data-efficient learning at web-scale through semantic deduplication, 2023 , author=. URL https://arxiv. org/abs/2303.09540 , year=

    Pith/arXiv arXiv 2023

  16. [16]

    CCF International Conference on Natural Language Processing and Chinese Computing , pages=

    Neural knowledge bank for pretrained transformers , author=. CCF International Conference on Natural Language Processing and Chinese Computing , pages=. 2023 , organization=

    2023

  17. [17]

    arXiv preprint arXiv:2210.08726 , year=

    Rarr: Researching and revising what language models say, using language models , author=. arXiv preprint arXiv:2210.08726 , year=

    Pith/arXiv arXiv

  18. [18]

    arXiv preprint arXiv:2310.10638 , year=

    In-context pretraining: Language modeling beyond document boundaries , author=. arXiv preprint arXiv:2310.10638 , year=

    Pith/arXiv arXiv

  19. [19]

    arXiv preprint arXiv:2309.03883 , year=

    Dola: Decoding by contrasting layers improves factuality in large language models , author=. arXiv preprint arXiv:2309.03883 , year=

    Pith/arXiv arXiv

  20. [20]

    arXiv preprint arXiv:2504.01990 , year=

    Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems , author=. arXiv preprint arXiv:2504.01990 , year=

    Pith/arXiv arXiv

  21. [21]

    arXiv preprint arXiv:2402.01680 , year=

    Large language model based multi-agents: A survey of progress and challenges , author=. arXiv preprint arXiv:2402.01680 , year=

    Pith/arXiv arXiv

  22. [22]

    A multi-agent assisted approach for qualitative data analysis , year=

    Can large language models serve as data analysts , author=. A multi-agent assisted approach for qualitative data analysis , year=

  23. [23]

    arXiv preprint arXiv:2401.02705 , year=

    Xuat-copilot: Multi-agent collaborative system for automated user acceptance testing with large language model , author=. arXiv preprint arXiv:2401.02705 , year=

    Pith/arXiv arXiv

  24. [24]

    arXiv preprint arXiv:2312.13010 , year=

    Agentcoder: Multi-agent-based code generation with iterative testing and optimisation , author=. arXiv preprint arXiv:2312.13010 , year=

    Pith/arXiv arXiv

  25. [25]

    arXiv preprint arXiv:2308.00352 , volume=

    Metagpt: Meta programming for multi-agent collaborative framework , author=. arXiv preprint arXiv:2308.00352 , volume=

    Pith/arXiv arXiv

  26. [26]

    arXiv preprint arXiv:2310.20151 , year=

    Multi-agent consensus seeking via large language models , author=. arXiv preprint arXiv:2310.20151 , year=

    Pith/arXiv arXiv

  27. [27]

    arXiv preprint arXiv:2310.07937 , year=

    Co-navgpt: Multi-robot cooperative visual semantic navigation using large language models , author=. arXiv preprint arXiv:2310.07937 , year=

    Pith/arXiv arXiv

  28. [28]

    arXiv preprint arXiv:2401.14589 , year=

    Enhancing diagnostic accuracy through multi-agent conversations: using large language models to mitigate cognitive bias , author=. arXiv preprint arXiv:2401.14589 , year=

    Pith/arXiv arXiv

  29. [29]

    Extreme Mechanics Letters , volume=

    MechAgents: Large language model multi-agent collaborations can solve mechanics problems, generate new data, and integrate knowledge , author=. Extreme Mechanics Letters , volume=. 2024 , publisher=

    2024

  30. [30]

    arXiv preprint arXiv:2503.01935 , year=

    MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents , author=. arXiv preprint arXiv:2503.01935 , year=

    Pith/arXiv arXiv

  31. [31]

    The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

    Can Large Language Model Agents Simulate Human Trust Behavior? , author=. The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

  32. [32]

    IEEE Wireless Communications , year=

    Large language model enhanced multi-agent systems for 6G communications , author=. IEEE Wireless Communications , year=

  33. [33]

    arXiv preprint arXiv:2312.00746 , year=

    Deciphering digital detectives: Understanding llm behaviors and capabilities in multi-agent mystery games , author=. arXiv preprint arXiv:2312.00746 , year=

    Pith/arXiv arXiv

  34. [34]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Can large language models serve as rational players in game theory? a systematic analysis , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  35. [35]

    Proceedings of the 47th international ACM SIGIR conference on research and development in Information Retrieval , pages=

    On generative agents in recommendation , author=. Proceedings of the 47th international ACM SIGIR conference on research and development in Information Retrieval , pages=

  36. [36]

    Proceedings of the ACM Web Conference 2024 , pages=

    Agentcf: Collaborative learning with autonomous language agents for recommender systems , author=. Proceedings of the ACM Web Conference 2024 , pages=

    2024

  37. [37]

    arXiv preprint arXiv:2406.11277 , year=

    Small agent can also rock! empowering small language models as hallucination detector , author=. arXiv preprint arXiv:2406.11277 , year=

    Pith/arXiv arXiv

  38. [38]

    arXiv preprint arXiv:2501.13946 , year=

    Hallucination Mitigation using Agentic AI Natural Language-Based Frameworks , author=. arXiv preprint arXiv:2501.13946 , year=

    Pith/arXiv arXiv

  39. [39]

    arXiv preprint arXiv:2503.18102 , year=

    Agentrxiv: Towards collaborative autonomous research , author=. arXiv preprint arXiv:2503.18102 , year=

    Pith/arXiv arXiv

  40. [40]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    When benchmarks are targets: Revealing the sensitivity of large language model leaderboards , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  41. [41]

    arXiv preprint arXiv:2501.07493 , year=

    Exploring and mitigating adversarial manipulation of voting-based leaderboards , author=. arXiv preprint arXiv:2501.07493 , year=

    Pith/arXiv arXiv

  42. [42]

    arXiv preprint arXiv:2504.20879 , year=

    The Leaderboard Illusion , author=. arXiv preprint arXiv:2504.20879 , year=

    Pith/arXiv arXiv

  43. [43]

    arXiv preprint arXiv:2410.21276 , year=

    Gpt-4o system card , author=. arXiv preprint arXiv:2410.21276 , year=

    Pith/arXiv arXiv

  44. [44]

    5 technical report , author=

    Qwen2. 5 technical report , author=. arXiv preprint arXiv:2412.15115 , year=

    Pith/arXiv arXiv

  45. [45]

    arXiv preprint arXiv:2412.19437 , year=

    Deepseek-v3 technical report , author=. arXiv preprint arXiv:2412.19437 , year=

    Pith/arXiv arXiv

  46. [46]

    arXiv preprint arXiv:2411.04368 , year=

    Measuring short-form factuality in large language models , author=. arXiv preprint arXiv:2411.04368 , year=

    Pith/arXiv arXiv

  47. [47]

    journal of Economic Theory , volume=

    Perfect Bayesian equilibrium and sequential equilibrium , author=. journal of Economic Theory , volume=. 1991 , publisher=

    1991

  48. [48]

    The eleventh international conference on learning representations , year=

    React: Synergizing reasoning and acting in language models , author=. The eleventh international conference on learning representations , year=

  49. [49]

    arXiv preprint arXiv:2509.17567 , year=

    Limi: Less is more for agency , author=. arXiv preprint arXiv:2509.17567 , year=

    arXiv

  50. [50]

    arXiv preprint arXiv:2508.06960 , year=

    Datasetresearch: Benchmarking agent systems for demand-driven dataset discovery , author=. arXiv preprint arXiv:2508.06960 , year=

    Pith/arXiv arXiv

  51. [51]

    arXiv preprint arXiv:2601.11044 , year=

    Agencybench: Benchmarking the frontiers of autonomous agents in 1m-token real-world contexts , author=. arXiv preprint arXiv:2601.11044 , year=

    Pith/arXiv arXiv

  52. [52]

    arXiv preprint arXiv:2510.27598 , year=

    InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Research , author=. arXiv preprint arXiv:2510.27598 , year=

    arXiv

  53. [53]

    arXiv preprint arXiv:2604.08963 , year=

    Aligned Agents, Biased Swarm: Measuring Bias Amplification in Multi-Agent Systems , author=. arXiv preprint arXiv:2604.08963 , year=

    Pith/arXiv arXiv

  54. [54]

    arXiv preprint arXiv:2602.02619 , year=

    daVinci-Agency: Unlocking Long-Horizon Agency Data-Efficiently , author=. arXiv preprint arXiv:2602.02619 , year=

    arXiv

  55. [55]

    European Conference on Computer Vision , pages=

    KAN-Mixer: Kolmogorov-Arnold Networks for Gene Expression Prediction in Plant Species , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024