Pith. sign in

REVIEW 3 major objections 3 minor 55 references

HIERA: Hierarchical Multi-Agent Relevance Assessment for Content Discovery Systems

T0 review · 3 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Hierarchical coordination, not extra computation, is what makes LLM relevance assessment accurate.

desk verdict A genuinely interesting multi-agent judge design, but the headline causal claim about coordination topology rests on a control that doesn't isolate the synthesis step. read the letter →

arxiv 2608.00785 v1 pith:5FHKD3J6 submitted 2026-08-01 cs.MA

Pritom Saha Akash , Phanideep Gampa , Chao Shen , Ying Chen , Sheikh Muhammad Sarwar This is my paper · ORCID
classification cs.MA
keywords relevanceassessmentmulti-agentLLMhierarchicalcoordinationcontentdiscoverysearchevaluationLLM-as-a-judgeexternalknowledgeintegrationablationstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the accuracy of LLM-based relevance assessment in search and product discovery depends less on how many models or how much context you use than on how you coordinate them. It proposes HIERA, a three-level hierarchy in which a Relevance Judge decides when specialist input is needed, a Relation Analyzer coordinates a Query Analyzer and an Item Analyzer plus external knowledge, and the Judge makes the final call on the synthesised argument. Across five datasets and eleven baselines, HIERA reports the best accuracy and agreement, with the largest gains on the proprietary EVS set (0.713 vs. 0.517) and consistent double-digit improvements over flat ensembles. The paper's distinctive causal claim is that these gains come from the coordination structure itself: the same agents and the same web knowledge, without hierarchical synthesis, score far lower at nearly the same token budget. If true, this reframes multi-agent evaluation as an architecture-design problem rather than a scaling problem.

What carries the argument

The load-bearing mechanism is the three-layer coordination graph: the Decision Layer's Relevance Judge ($J$) decides whether to delegate; the Analysis Layer's Relation Analyzer ($R$) consults the Query Analyzer ($Q$) and Item Analyzer ($I$) as needed and integrates their findings with Information Layer knowledge ($K$) into one structured relevance argument; the Judge then produces the final label from that argument. $R$ is the central synthesizer that distinguishes the architecture from flat aggregation: specialist outputs pass through $R$ and are interleaved with its own connection analysis before the Judge sees them. The paper defines the Judge's delegation trigger as three evidence dimensions—query interpretability, information completeness, and relational transparency—and reports that the full hierarchy is activated in 91.5% of cases.

What would settle it

Run HIERA and the 'Judge + All Analyzers' variant on MSRD with verbatim prompt logs, matched per-agent token limits, the same web-search results, and identical specialist outputs; if the accuracy gap (0.859 vs. 0.762) collapses or fails McNemar's test at $p<0.05$ when prompts and compute are exactly matched, the coordination-structure claim is falsified. A second check is to feed the Judge the exact final argument produced by the Relation Analyzer as plain text in the uncoordinated variant; if that closes most of the gap, the effect comes from the content of synthesis rather than the interactive routing.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that HIERA outperforms single-step, multi-step, and ensemble LLM relevance assessors on all five evaluation datasets, and that the advantage survives an ablation that fixes the agents and knowledge. The full system reaches 0.859 accuracy on MSRD and 0.681 on WANDS, while the uncoordinated 'Judge + All Analyzers' variant—same specialist outputs, same web search, orchestration-level tokens within 1.8%—reaches only 0.762 and 0.659. The same external knowledge that improves HIERA by 12.7% on MSRD degrades most baselines when concatenated directly into their prompts. From this the authors conclude that the Relation Analyzer's interactive synthesis, not additional computation or context, is the active ingredient, and that coordination topology determines whether adding agents and knowledge helps or hurts.

Load-bearing premise

The load-bearing premise is that the uncoordinated 'Judge + All Analyzers' ablation is a fair control—identical specialist outputs, web knowledge, model versions, prompt templates, and token budgets, differing only by the absence of the Relation Analyzer's synthesis—so the 12.7% accuracy gap can be attributed to coordination rather than to prompt phrasing, output ordering, or unmeasured compute.

Editorial extensions

If this is right

  • Practitioners should expect the largest gains on cases that require connecting separate dimensions, such as implicit color equivalence or style compatibility, where flat and staged baselines systematically underestimate relevance.
  • Adding web knowledge or extra specialist analyses to an LLM judge can reduce accuracy unless a coordinating layer synthesises them; knowledge integration belongs in the coordination protocol, not appended to prompts.
  • Multi-agent evaluation comparisons should hold compute and context constant: the paper's topologies differ by only 1.8% in orchestration-level tokens yet differ by 37% in Cohen's $\kappa$ (0.524 vs. 0.718 on MSRD).
  • For offline evaluation workloads, the 35-second latency is claimed acceptable because it remains 50 times faster than human experts, so the accuracy gains come at a latency cost that rules out real-time use but suits offline labeling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same coordination principle should transfer to other LLM families and languages, which the paper does not test; a direct replication with GPT or Gemini models on the public datasets would show whether the hierarchy benefit is Claude-specific.
  • Editorial inference: because the Judge delegates in 91.5% of cases, HIERA is not acting as an adaptive router; a cheaper variant that predicts delegability and runs the full hierarchy only on hard cases could preserve most of the gain at lower cost, an optimisation the paper leaves open.
  • Editorial inference: the two documented failure modes suggest targeted fixes—a fact-checking step that prevents specialists from reinforcing an unsupported cultural association, and a softer partial-relevance option when one constraint mismatches but the category matches; both are testable modifications rather than new frameworks.
  • Editorial inference: if the synthesis effect is truly about information ordering and interactive consultation, a single LLM given the same final structured argument as a chain-of-thought should close part of the gap; that experiment would isolate the benefit of multi-agent routing from the benefit of the reasoning trace itself.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes HIERA, a three-layer hierarchical multi-agent framework for LLM-based relevance assessment in content discovery. A Relevance Judge decides when specialist consultation is needed and delegates to a Relation Analyzer, which coordinates a Query Analyzer, an Item Analyzer, and external knowledge access before returning a synthesized relevance argument for final judgment. The authors report consistent accuracy/F1/Cohen's-κ improvements over 11 baselines across five datasets, with headline gains on a proprietary EVS set (0.713 vs. 0.517 accuracy) and on public datasets. Ablation studies on MSRD and WANDS are used to argue that the coordination structure itself, rather than additional context or token budget, drives the improvements, including a 12.7% gain over an uncoordinated 'Judge + All Analyzers' variant. The paper includes complete system prompt templates, bootstrap confidence intervals for ablations, and qualitative case studies of successes and failures.

Significance. If the central claim holds, the paper makes a useful contribution: it demonstrates that a hierarchy of specialized LLM agents with an explicit synthesis step can outperform flat prompting, staged pipelines, and ensemble aggregation for relevance judgment, and that simply adding agents or external knowledge without coordination can hurt. The work is empirically broad (five datasets, eleven baselines), and the appendix provides full prompt templates and bootstrap CIs for the ablations, which supports reproducibility. The strongest finding—that coordination topology matters beyond token count—is thought-provoking and would have practical implications for multi-agent evaluation systems. However, the causal attribution to 'coordination structure' is currently under-tested because the key ablation does not hold the synthesis step constant, and the main headline result on EVS relies on a small proprietary set without reported confidence intervals.

major comments (3)
  1. [Section 6.2, Table 3; Section 7.3] The causal claim that hierarchical coordination rather than an additional synthesis inference drives the 12.7% gain (MSRD κ 0.718 vs. 0.524) is not adequately supported. In full HIERA, R is explicitly prompted to 'Provide comprehensive relationship analysis' and synthesize findings into a structured relevance argument (Appendix B.2), while the 'Judge + All Analyzers' ablation appears to consist of J receiving the specialist outputs as context with no explicit synthesis step. The paper reports only orchestration-level token counts (2,917 vs. 2,866) and does not report the number of LLM calls, hidden ReAct reasoning tokens, or the exact prompt used for the uncoordinated control. Consequently, the gap could be explained by an extra serial synthesis inference over all specialist outputs, which is orthogonal to hierarchical routing. Please provide a control that holds the synthesis step constant (e.g., add an explicit synthesis prompt to the uncoordinated condition, or remove R's synthesis role from HIERA) and report per-step token and inference counts.
  2. [Tables 1 and 2; Section 5.3] The main results tables lack confidence intervals, and the headline gain on EVS is based on a proprietary 300-pair set. The text states 'substantial and consistent improvements across all evaluation datasets' but only the ablation conditions in Appendix D report bootstrap CIs. Without CIs for the main comparisons, it is difficult to assess the reliability of the 38% EVS accuracy gain and the smaller gains on ESCI and Home Depot where the absolute accuracy is low (e.g., 0.464 on Home Depot). Please add bootstrap CIs (or equivalent uncertainty measures) for all main results, and clarify whether the balanced ~1,000-pair subsamples are used for the numbers in Tables 1 and 2 or the full datasets.
  3. [Section 5.1] The evaluation design uses balanced subsamples of approximately 1,000 query-item pairs per public dataset, with equal representation of each relevance class, but the paper does not justify this choice or demonstrate that results transfer to the original class distribution. Accuracy and macro-F1 are sensitive to class priors; for example, some baselines have very low F1 on Home Depot (e.g., Multi-Stage at 0.285), which may be an artifact of the balanced subsample rather than the underlying dataset difficulty. Please report results on the full datasets (or a clearly justified random subsample) and compare against the original label distribution, or argue explicitly why the balanced subsample is the appropriate evaluation setting.
minor comments (3)
  1. [Throughout] There are several typographical issues: 'W ANDS' appears with a stray space in Table 2 and in the text, and the abstract has 'p<0.05 )' with an extra space before the parenthesis. These should be corrected.
  2. [References] Several references contain formatting errors, such as 'V oorhees' (Voorhees) and 'Sanderson and 1 others' in the reference list. Please standardize author names and ensure the bibliography is clean.
  3. [Appendix B and C] The prompt templates for Q and I in Appendix B are nearly identical except for the task description; the paper does not explain how the 'Focus Areas' parameter varies across datasets, which is important for reproducing the specialization. Please provide the actual instantiated focus areas for each dataset.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: HIERA's claims rest on external ground-truth evaluation and empirical ablations, not on inputs that reduce to the predicted conclusion.

full rationale

The paper contains no derivation chain in which an output is equivalent to an input by construction. The central claim—that hierarchical coordination through the Relation Analyzer improves relevance assessment—is supported by comparisons against external ground-truth datasets (EVS, MSRD, ESCI, WANDS, Home Depot) and by ablation studies that vary the architecture while keeping agents and knowledge access fixed. The ablation 'Judge + All Analyzers' versus full HIERA is an empirical control, not a restatement of the hypothesis; the reported 12.7% gap is measured, not fitted. The coordination equations (1)–(6) merely describe the communication protocol and are not used to derive performance. There are no fitted parameters renamed as predictions, no uniqueness theorem invoked from the authors' own prior work, and no load-bearing self-citations: the paper's references point to external methods and datasets. A legitimate concern is that the causal claim about coordination topology may be confounded by the extra synthesis pass performed by R, and that some design choices may have been selected using the same evaluation sets without a validation split; however, these are experimental-validity and selection issues, not circular reasoning. The paper is self-contained against external benchmarks, so the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

HIERA introduces no new physical or theoretical entities; the four agents are software components, and their 'independent evidence' is the benchmark evaluation inside the paper rather than an external falsifiable handle. The central claim rests on domain assumptions about label quality, the decomposition of relevance into three dimensions, the accuracy of web-search knowledge, and the representativeness of balanced subsamples. The only numeric hand-chosen settings are web-search top-k, token limits, model assignment, temperature, and the subsample construction, none of which is swept in a sensitivity analysis.

free parameters (4)
  • Evaluation subset size and class balance = ~1,000 pairs per dataset; equal representation per relevance level; seed 42
    Section 5.1: subsets are random samples with fixed seed and equal relevance-level representation. The choice changes class priors, affecting macro F1 and kappa, and is not justified against full-dataset evaluation.
  • Web search top-k = 5 results per query
    Appendix A.1: 'Web search returns 5 results per query.' Hand-chosen constant; knowledge integration results in Table 4 depend on this.
  • Agent model assignment and token limits = J/R: Claude 3.7 Sonnet (3000/1500 max tokens); Q/I: Claude 3.5 Haiku (1000 max tokens)
    Appendix A.1. These choices affect reasoning capacity and cost; no sensitivity analysis is reported.
  • Temperature = 0
    Set for deterministic outputs (Section 5.4). The authors note decision stability across runs is not formally evaluated.
assumptions (5)
  • domain assumption Ground-truth labels in EVS, MSRD, ESCI, WANDS, and Home Depot are correct and reflect searcher intent.
    Section 5.1 treats dataset labels as ground truth for accuracy, F1, and kappa; any label noise directly changes all reported gains.
  • ad hoc to paper Relevance can be decomposed into query interpretability, item information, and relational transparency, and the four LLM agents can perform these subtasks reliably.
    Section 4.2 defines the three dimensions and assigns them to agents; if this decomposition is wrong, the framework's advantage would not generalize.
  • domain assumption Web search results used by agents are accurate and sufficient for establishing relationships such as color equivalence and style compatibility.
    Section 4.2.3 and Appendix A.1; web search top-5 results are assumed to provide correct external knowledge, with no verification of retrieval quality.
  • ad hoc to paper Balanced ~1,000-pair subsamples are representative of the full datasets, and author re-implementations of 11 baselines are faithful to the original papers.
    Section 5.1 and Appendix A.2. Both assumptions are untested; a non-representative subsample or a weak baseline implementation would inflate HIERA's relative gains.
  • domain assumption LLM outputs at temperature 0 are stable enough for paired significance testing.
    Section 5.4 and Limitations: the authors do not formally evaluate decision stability across repeated runs or model versions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HIERA: Hierarchical Multi-Agent Relevance Assessment for Content Discovery Systems." pith.science (2026). https://pith.science/paper/5FHKD3J6

@misc{pith2026260800785,
  author       = {Pith},
  title        = {Pith review of: HIERA: Hierarchical Multi-Agent Relevance Assessment for Content Discovery Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5FHKD3J6}},
  note         = {Machine review of arXiv:2608.00785}
}
abstract

Content discovery systems depend on relevance judgment for search quality evaluation, but human annotation faces inter-annotator disagreement and scaling costs. While Large Language Models show promise as automated assessors, current approaches rely on flat aggregation strategies: single-step prompting, voting ensembles, or uncoordinated multi-agent pipelines that aggregate independent outputs without integration. We propose HIERA, a hierarchical multi-agent relevance assessment framework with four specialized agents: a Relevance Judge, Query Analyzer, Item Analyzer, and Relation Analyzer. The Judge determines when specialist analysis is needed; the Relation Analyzer then coordinates query and item analyses with external knowledge to establish relevance relationships for final judgment. Ablation studies show that the same agents and external knowledge without hierarchical coordination degrade performance, confirming that the coordination structure itself accounts for the improvement. Evaluation across five datasets (EVS, MSRD, ESCI, WANDS, Home Depot) shows improvements over 11 baselines: 10.2\% on Home Depot, 4.8\% on ESCI, and up to 38\% on EVS ($p < 0.05$). Hierarchical coordination yields 12.7\% improvement over uncoordinated collaboration using identical agents.

Figures

Figures reproduced from arXiv: 2608.00785 by the authors.

Figure 1
Figure 1. HIERA vs. other LLM-based approaches. two challenges. First, relevance is inherently sub￾jective, with annotators frequently disagreeing on labels (Voorhees, 1998; Bailey et al., 2008). Sec￾ond, annotating thousands of query-item pairs across domains is economically infeasible (Thomas et al., 2024), forcing systems to operate with lim￾ited ground truth (Reddy et al., 2022; Deldjoo et al., 2020). To address these cha… view at source ↗
Figure 2
Figure 2. HIERA Framework coordinate through a three-layer hierarchy where information flows between layers through tool￾based consultation: Decision Layer : J → {R} (1) Analysis Layer : [R ↔ {Q, I}] → K (2) Information Layer : K (3) This hierarchical design enables structured evi￾dence synthesis: each specialist’s output informs subsequent analytical steps, and the Relation Ana￾lyzer integrates findings into a coherent relev… view at source ↗
Figure 3
Figure 3. Judge Agent prompt template. chitecture for full reproducibility. Each template serves as the system prompt for the respective agent, defining its role, capabilities, and behav￾ioral constraints within the multi-agent coordina￾tion framework. All prompts utilize configurable parameters (shown in red) that are dynamically populated based on domain-specific configurations and dataset characteristics. B.1 Judge Agent S… view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Query Analyzer prompt template. stability [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Item Analyzer prompt template. Instantiated Judge Prompt (WANDS) TASK: Judge relevance between query and candidate result DOMAIN: Home Furnishings and Decor Search Home furnishings and decor search involves finding furniture and decora￾tive items that match customer ho…
Figure 7
Figure 7. Figure 7: Instantiated Judge prompt for WANDS dataset. from initial assessment → semantic analysis → external validation → final decision, building ev￾idence for language constraint violations. Case 2 shows functional analysis → category verifica￾tion → purpose differentiation, …
Figure 8
Figure 8. Figure 8: HIERA Success Cases: Hierarchical reason [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: HIERA Error Cases: False positive and false [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 26 canonical work pages

  1. [1]

    Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

    One-shot labeling for automatic relevance estimation , author=. Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

    work page

  2. [2]

    Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval , pages=

    Perspectives on large language models for relevance judgment , author=. Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval , pages=

    work page 2023

  3. [3]

    arXiv preprint arXiv:2411.08275 , year=

    A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look , author=. arXiv preprint arXiv:2411.08275 , year=

    arXiv

  4. [4]

    Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region , pages=

    Ai can be cognitively biased: An exploratory study on threshold priming in llm-based batch relevance assessment , author=. Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region , pages=

    work page 2024

  5. [5]

    Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

    Large language models can accurately predict searcher preferences , author=. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

    work page

  6. [6]

    arXiv preprint arXiv:2408.08896 , year=

    Llmjudge: Llms for relevance judgments , author=. arXiv preprint arXiv:2408.08896 , year=

    arXiv

  7. [7]

    LLMs can be Fooled into Labelling a Document as Relevant: best caf

    Alaofi, Marwah and Thomas, Paul and Scholer, Falk and Sanderson, Mark , booktitle=. LLMs can be Fooled into Labelling a Document as Relevant: best caf

    work page

  8. [8]

    arXiv preprint arXiv:2405.04727 , year=

    Llms can patch up missing relevance judgments in evaluation , author=. arXiv preprint arXiv:2405.04727 , year=

    arXiv

Show all 55 references
  1. [9]

    Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

    Synthetic test collections for retrieval evaluation , author=. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

  2. [10]

    arXiv preprint arXiv:2406.06519 , year=

    Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor , author=. arXiv preprint arXiv:2406.06519 , year=

    arXiv

  3. [11]

    arXiv preprint arXiv:2504.09816 , year=

    Augmented Relevance Datasets with Fine-Tuned Small LLMs , author=. arXiv preprint arXiv:2504.09816 , year=

    arXiv

  4. [12]

    arXiv preprint arXiv:2504.12558 , year=

    Benchmarking LLM-based Relevance Judgment Methods , author=. arXiv preprint arXiv:2504.12558 , year=

    arXiv

  5. [13]

    Companion Proceedings of the ACM on Web Conference 2025 , pages=

    JudgeBlender: Ensembling Automatic Relevance Judgments , author=. Companion Proceedings of the ACM on Web Conference 2025 , pages=

    2025

  6. [14]

    arXiv preprint arXiv:2502.13908 , year=

    Judging the judges: A collection of llm-generated relevance judgements , author=. arXiv preprint arXiv:2502.13908 , year=

    arXiv

  7. [15]

    arXiv preprint arXiv:2504.05220 , year=

    Leveraging LLMs for Utility-Focused Annotation: Reducing Manual Effort for Retrieval and RAG , author=. arXiv preprint arXiv:2504.05220 , year=

  8. [16]

    arXiv preprint arXiv:2411.06877 , year=

    LLM-Assisted Relevance Assessments: When Should We Ask LLMs for Help? , author=. arXiv preprint arXiv:2411.06877 , year=

    arXiv

  9. [17]

    arXiv preprint arXiv:2412.17156 , year=

    LLM-based relevance assessment still can't replace human relevance assessment , author=. arXiv preprint arXiv:2412.17156 , year=

  10. [18]

    arXiv preprint arXiv:2504.19076 , year=

    LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluations , author=. arXiv preprint arXiv:2504.19076 , year=

  11. [19]

    Companion Proceedings of the ACM on Web Conference 2025 , pages=

    Multi-stage large language model pipelines can outperform gpt-4o in relevance assessment , author=. Companion Proceedings of the ACM on Web Conference 2025 , pages=

    2025

  12. [20]

    ACM Transactions on Information Systems , year=

    Query performance prediction using relevance judgments generated by large language models , author=. ACM Transactions on Information Systems , year=

  13. [21]

    arXiv preprint arXiv:2503.19092 , year=

    Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation , author=. arXiv preprint arXiv:2503.19092 , year=

    arXiv

  14. [22]

    European Conference on Information Retrieval , pages=

    ChatGPT goes shopping: LLMs can predict relevance in ecommerce search , author=. European Conference on Information Retrieval , pages=. 2024 , organization=

    2024

  15. [23]

    arXiv preprint arXiv:2406.00247 , year=

    Large language models for relevance judgment in product search , author=. arXiv preprint arXiv:2406.00247 , year=

    arXiv

  16. [24]

    Proceedings of the 2024 8th International Conference on Natural Language Processing and Information Retrieval , pages=

    Automated Query-Product Relevance Labeling using Large Language Models for E-commerce Search , author=. Proceedings of the 2024 8th International Conference on Natural Language Processing and Information Retrieval , pages=

    2024

  17. [25]

    European Conference on Information Retrieval , pages=

    Retrieve, Annotate, Evaluate, Repeat: Leveraging Multimodal LLMs for Large-Scale Product Retrieval Evaluation , author=. European Conference on Information Retrieval , pages=. 2025 , organization=

    2025

  18. [26]

    Enhancing LLM-as-a-judge via multi-agent collaboration , author=

  19. [27]

    Reddy and Lluís Màrquez and Fran Valero and Nikhil Rao and Hugo Zaragoza and Sambaran Bandyopadhyay and Arnab Biswas and Anlu Xing and Karthik Subbian , year=

    Chandan K. Reddy and Lluís Màrquez and Fran Valero and Nikhil Rao and Hugo Zaragoza and Sambaran Bandyopadhyay and Arnab Biswas and Anlu Xing and Karthik Subbian , year=. Shopping Queries Dataset: A Large-Scale. 2206.06588 , archivePrefix=

    arXiv

  20. [28]

    Proceedings of the 44th European Conference on Information Retrieval , year =

    WANDS: Dataset for Product Search Relevance Assessment , author =. Proceedings of the 44th European Conference on Information Retrieval , year =

  21. [29]

    ACM Transactions on Information Systems (TOIS) , volume=

    Cumulated gain-based evaluation of IR techniques , author=. ACM Transactions on Information Systems (TOIS) , volume=. 2002 , publisher=

    2002

  22. [30]

    2008 , publisher=

    Introduction to information retrieval , author=. 2008 , publisher=

    2008

  23. [31]

    Foundations and Trends

    Test collection based evaluation of information retrieval systems , author=. Foundations and Trends. 2010 , publisher=

    2010

  24. [32]

    Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval , pages=

    Variations in relevance judgments and the measurement of retrieval effectiveness , author=. Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval , pages=

  25. [33]

    Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval , pages=

    Relevance assessment: are judges exchangeable and does it matter , author=. Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval , pages=

  26. [34]

    ACM Computing Surveys (CSUR) , volume=

    Recommender systems leveraging multimedia content , author=. ACM Computing Surveys (CSUR) , volume=. 2020 , publisher=

    2020

  27. [35]

    ACM Sigir Forum , volume=

    Challenges and research opportunities in ecommerce search and recommendations , author=. ACM Sigir Forum , volume=. 2021 , organization=

    2021

  28. [36]

    arXiv preprint arXiv:2410.14044 , year=

    Best in tau@ llmjudge: Criteria-based relevance evaluation with llama3 , author=. arXiv preprint arXiv:2410.14044 , year=

    arXiv

  29. [37]

    Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR) , pages=

    Criteria-Based LLM Relevance Judgments , author=. Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR) , pages=

    2025

  30. [38]

    arXiv preprint arXiv:2410.10934 , year=

    Agent-as-a-judge: Evaluate agents with agents , author=. arXiv preprint arXiv:2410.10934 , year=

    arXiv

  31. [39]

    arXiv preprint arXiv:2504.17087 , year=

    Leveraging llms as meta-judges: A multi-agent framework for evaluating llm judgments , author=. arXiv preprint arXiv:2504.17087 , year=

    arXiv

  32. [40]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    SQLFixAgent: Towards Semantic-Accurate Text-to-SQL Parsing via Consistency-Enhanced Multi-Agent Collaboration , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  33. [41]

    arXiv preprint arXiv:2504.19678 , year=

    From llm reasoning to autonomous ai agents: A comprehensive review , author=. arXiv preprint arXiv:2504.19678 , year=

    arXiv

  34. [42]

    International Conference on Learning Representations (ICLR) , year=

    React: Synergizing reasoning and acting in language models , author=. International Conference on Learning Representations (ICLR) , year=

  35. [43]

    arXiv preprint arXiv:2305.16291 , year=

    Voyager: An open-ended embodied agent with large language models , author=. arXiv preprint arXiv:2305.16291 , year=

    arXiv

  36. [44]

    First Conference on Language Modeling , year=

    Autogen: Enabling next-gen LLM applications via multi-agent conversations , author=. First Conference on Language Modeling , year=

  37. [45]

    arXiv preprint arXiv:2308.10848 , volume=

    Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents , author=. arXiv preprint arXiv:2308.10848 , volume=

    arXiv

  38. [46]

    2023 , url=

    AutoGPT: Build, Deploy, and Run AI Agents , author=. 2023 , url=

    2023

  39. [47]

    The Twelfth International Conference on Learning Representations , year=

    MetaGPT: Meta programming for a multi-agent collaborative framework , author=. The Twelfth International Conference on Learning Representations , year=

  40. [48]

    arXiv preprint arXiv:2404.17153 , year=

    A unified debugging approach via llm-based multi-agent synergy , author=. arXiv preprint arXiv:2404.17153 , year=

  41. [49]

    arXiv preprint arXiv:2312.11242 , year=

    Mac-sql: A multi-agent collaborative framework for text-to-sql , author=. arXiv preprint arXiv:2312.11242 , year=

    arXiv

  42. [50]

    arXiv preprint arXiv:2403.11886 , year=

    Queryagent: A reliable and efficient reasoning framework with environmental feedback-based self-correction , author=. arXiv preprint arXiv:2403.11886 , year=

    arXiv

  43. [51]

    arXiv preprint arXiv:2505.16944 , year=

    Agentif: Benchmarking instruction following of large language models in agentic scenarios , author=. arXiv preprint arXiv:2505.16944 , year=

    arXiv

  44. [52]

    arXiv preprint arXiv:2504.09037 , year=

    A survey of frontiers in llm reasoning: Inference scaling, learning to reason, and agentic systems , author=. arXiv preprint arXiv:2504.09037 , year=

  45. [53]

    arXiv preprint arXiv:2308.07201 , year=

    Chateval: Towards better llm-based evaluators through multi-agent debate , author=. arXiv preprint arXiv:2308.07201 , year=

    arXiv

  46. [54]

    arXiv preprint arXiv:2402.01135 , year=

    A multi-agent conversational recommender system , author=. arXiv preprint arXiv:2402.01135 , year=

    arXiv

  47. [55]

    Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

    Macrec: A multi-agent collaboration framework for recommendation , author=. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.