{"id":"d91191cb-15ce-40bf-94a5-a7eee0e0d753","arxiv_id":"2608.00785","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A hierarchical multi-agent LLM framework, in which a coordinating analyzer synthesizes query and item specialist outputs, beats flat, staged, and ensemble LLM relevance judges on five content search datasets.","lead":"HIERA is a multi-agent system that splits relevance judgment into a Judge, a Relation Analyzer, and query and item specialists, with the Relation Analyzer synthesizing specialist findings before the final call. Across five search datasets it reports higher accuracy than 11 single-step, staged, and ensemble LLM baselines, and its ablations argue that coordination structure, not added information, drives the gains.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The causal claim in §6.2 is under-tested: the 'Judge + All Analyzers' control lacks R's explicit synthesis pass, so the 12.7% gain may reflect an extra synthesis step or hidden compute, not hierarchical coordination topology.","rationale":"The reader's weakest_assumption already pointed at control fairness; my check sharpens it to a specific confound: the uncoordinated control may omit any structured-synthesis stage, so the comparison is 'hierarchy plus synthesis' versus 'no synthesis'. The authors' token-budget matching at orchestration level does not rule out extra serial inference: token counts are not equivalent to compute steps or to the presence of an integrative prompt. This is a real soft spot because the paper's strongest claim is causal ('coordination structure itself accounts for the improvement', §6.2). The proposed flat-synthesis control would settle it cleanly. I do not see grounds for REJECT: the method is described in detail, the ablation design is thoughtful, and the main results are internally consistent. CONDITIONAL (as the reader concluded) remains the right verdict until the control is tightened or code and full prompts are released.","tokens_in":15317,"tokens_out":6036,"duration_ms":58268,"concrete_test":"Run a flat-synthesis control on MSRD and WANDS using the same models and budgets: execute Q and I with web search independently, then pass their outputs plus web results to one Claude 3.7 synthesis call with R's prompt and 1,500-token budget to produce a structured relevance argument, and finally have J judge from that argument. Compare accuracy and κ against HIERA and against Judge+All Analyzers, with bootstrap CIs. If flat synthesis matches HIERA (MSRD κ≈0.718, WANDS κ≈0.521), the §6.2 causal claim fails; if it stays near the uncoordinated ablation (κ≈0.522 and 0.486), the topology explanation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that coordination topology, not extra computation or richer context, causes HIERA's gains—rests on the §6.2 ablation comparing full HIERA with 'Judge + All Analyzers.' That control is not symmetry-preserving. In full HIERA, R receives Q and I outputs plus web evidence and is explicitly prompted (Appendix B.2) to 'provide comprehensive relationship analysis' and synthesize findings into a structured relevance argument before J judges. In the uncoordinated control, no prompt shown in Appendix B or C instructs any agent to integrate Q's and I's outputs into a shared synthesis; J only receives analyses as context. The paper reports only orchestration-level token counts (§7.3: 2,917 vs 2,866 tokens), not the number of LLM calls, hidden ReAct reasoning tokens, or whether the control contains an equivalent synthesis pass. Consequently, the persistent 12.7% gap (MSRD κ 0.718 vs 0.524) could be explained by one additional serial synthesis inference over all specialist outputs, which is orthogonal to hierarchical routing. The causal conclusion needs a control that holds the synthesis step constant.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HIERA, a three-layer hierarchical multi-agent framework for LLM-based relevance assessment in content discovery. A Relevance Judge decides when specialist consultation is needed and delegates to a Relation Analyzer, which coordinates a Query Analyzer, an Item Analyzer, and external knowledge access before returning a synthesized relevance argument for final judgment. The authors report consistent accuracy/F1/Cohen's-κ improvements over 11 baselines across five datasets, with headline gains on a proprietary EVS set (0.713 vs. 0.517 accuracy) and on public datasets. Ablation studies on MSRD and WANDS are used to argue that the coordination structure itself, rather than additional context or token budget, drives the improvements, including a 12.7% gain over an uncoordinated 'Judge + All Analyzers' variant. The paper includes complete system prompt templates, bootstrap confidence intervals for ablations, and qualitative case studies of successes and failures.","tokens_in":15604,"tokens_out":3547,"duration_ms":32322,"significance":"If the central claim holds, the paper makes a useful contribution: it demonstrates that a hierarchy of specialized LLM agents with an explicit synthesis step can outperform flat prompting, staged pipelines, and ensemble aggregation for relevance judgment, and that simply adding agents or external knowledge without coordination can hurt. The work is empirically broad (five datasets, eleven baselines), and the appendix provides full prompt templates and bootstrap CIs for the ablations, which supports reproducibility. The strongest finding—that coordination topology matters beyond token count—is thought-provoking and would have practical implications for multi-agent evaluation systems. However, the causal attribution to 'coordination structure' is currently under-tested because the key ablation does not hold the synthesis step constant, and the main headline result on EVS relies on a small proprietary set without reported confidence intervals.","major_comments":[{"comment":"The causal claim that hierarchical coordination rather than an additional synthesis inference drives the 12.7% gain (MSRD κ 0.718 vs. 0.524) is not adequately supported. In full HIERA, R is explicitly prompted to 'Provide comprehensive relationship analysis' and synthesize findings into a structured relevance argument (Appendix B.2), while the 'Judge + All Analyzers' ablation appears to consist of J receiving the specialist outputs as context with no explicit synthesis step. The paper reports only orchestration-level token counts (2,917 vs. 2,866) and does not report the number of LLM calls, hidden ReAct reasoning tokens, or the exact prompt used for the uncoordinated control. Consequently, the gap could be explained by an extra serial synthesis inference over all specialist outputs, which is orthogonal to hierarchical routing. Please provide a control that holds the synthesis step constant (e.g., add an explicit synthesis prompt to the uncoordinated condition, or remove R's synthesis role from HIERA) and report per-step token and inference counts.","section":"Section 6.2, Table 3; Section 7.3"},{"comment":"The main results tables lack confidence intervals, and the headline gain on EVS is based on a proprietary 300-pair set. The text states 'substantial and consistent improvements across all evaluation datasets' but only the ablation conditions in Appendix D report bootstrap CIs. Without CIs for the main comparisons, it is difficult to assess the reliability of the 38% EVS accuracy gain and the smaller gains on ESCI and Home Depot where the absolute accuracy is low (e.g., 0.464 on Home Depot). Please add bootstrap CIs (or equivalent uncertainty measures) for all main results, and clarify whether the balanced ~1,000-pair subsamples are used for the numbers in Tables 1 and 2 or the full datasets.","section":"Tables 1 and 2; Section 5.3"},{"comment":"The evaluation design uses balanced subsamples of approximately 1,000 query-item pairs per public dataset, with equal representation of each relevance class, but the paper does not justify this choice or demonstrate that results transfer to the original class distribution. Accuracy and macro-F1 are sensitive to class priors; for example, some baselines have very low F1 on Home Depot (e.g., Multi-Stage at 0.285), which may be an artifact of the balanced subsample rather than the underlying dataset difficulty. Please report results on the full datasets (or a clearly justified random subsample) and compare against the original label distribution, or argue explicitly why the balanced subsample is the appropriate evaluation setting.","section":"Section 5.1"}],"minor_comments":[{"comment":"There are several typographical issues: 'W ANDS' appears with a stray space in Table 2 and in the text, and the abstract has 'p<0.05 )' with an extra space before the parenthesis. These should be corrected.","section":"Throughout"},{"comment":"Several references contain formatting errors, such as 'V oorhees' (Voorhees) and 'Sanderson and 1 others' in the reference list. Please standardize author names and ensure the bibliography is clean.","section":"References"},{"comment":"The prompt templates for Q and I in Appendix B are nearly identical except for the task description; the paper does not explain how the 'Focus Areas' parameter varies across datasets, which is important for reproducing the specialization. Please provide the actual instantiated focus areas for each dataset.","section":"Appendix B and C"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Ken,\n\nQuick take: HIERA is worth reading. The contribution is concrete: a Relation Analyzer that sits between the Judge and the Query/Item specialists, explicitly synthesizes their outputs into a relevance argument, and then hands it back to the Judge. That configuration is new relative to the single-step, multi-stage, and ensemble LLM judges in the cited literature, and the authors make a real case for it with five datasets, eleven baselines, and a set of ablations that control for most of the obvious confounds. The gray-dresser example is a good illustration of why flat approaches fail on multi-dimensional relevance.\n\nThe paper is honest about its limitations. They report token budgets, include failure cases where the hierarchy reinforces a misinterpretation, and acknowledge the EVS dataset is proprietary. The finding that adding web knowledge to most baselines degrades them while helping HIERA is a nice point in favor of coordination.\n\nThe soft spot is exactly the one the stress-test flags. The 'Judge + All Analyzers' ablation removes R's synthesis pass. Full HIERA has R explicitly prompted to 'provide comprehensive relationship analysis' and integrate findings into a structured argument. The control appears to just give J the specialist analyses as context, with no equivalent synthesis step. So the 12.7 percentage-point gap could come from an extra reasoning-inference over all outputs, rather than from the hierarchical routing itself. The paper reports orchestration-level token counts (2,917 vs 2,866) but not the number of LLM calls or hidden ReAct tokens, so the 'same compute' claim is not established. This is fixable: hold the synthesis step constant by having R (or an equivalent synthesizer) in both conditions, or by adding a flat synthesis pass to the control.\n\nMinor issues: the main tables lack confidence intervals; only ablations have bootstrap CIs. The EVS gain is on 300 proprietary pairs. The evaluation uses balanced ~1,000-pair subsamples, which is fine but not the full public datasets. No code or data released, and the baselines are reimplementations. None of these are fatal; they just limit the strength of the causal conclusions.\n\nVerdict: this deserves a serious referee. The design is clear, the experiments are broad, and the ablation is a genuine attempt at isolating the mechanism. It would be a useful addition to the LLM-as-judge literature after the control is tightened and the prompts/orderings are released.\n\nFor you: worth a read if you work on automated relevance judgment or multi-agent LLM evaluation. I'd cite it as related work, but I wouldn't yet rely on the 'coordination topology' claim as established.\n\nBest.","headline":"A genuinely interesting multi-agent judge design, but the headline causal claim about coordination topology rests on a control that doesn't isolate the synthesis step.","tokens_in":16081,"tokens_out":2957,"would_cite":true,"duration_ms":25693,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hierarchical coordination, not extra computation, is what makes LLM relevance assessment accurate.","keywords":["relevance assessment","multi-agent LLM","hierarchical coordination","content discovery","search evaluation","LLM-as-a-judge","external knowledge integration","ablation study"],"falsifier":"Run HIERA and the 'Judge + All Analyzers' variant on MSRD with verbatim prompt logs, matched per-agent token limits, the same web-search results, and identical specialist outputs; if the accuracy gap (0.859 vs. 0.762) collapses or fails McNemar's test at $p<0.05$ when prompts and compute are exactly matched, the coordination-structure claim is falsified. A second check is to feed the Judge the exact final argument produced by the Relation Analyzer as plain text in the uncoordinated variant; if that closes most of the gap, the effect comes from the content of synthesis rather than the interactive routing.","tokens_in":15111,"feed_emoji":"🤖","tokens_out":8714,"duration_ms":66233,"temperature":0.7,"pith_summary":"The paper claims that the accuracy of LLM-based relevance assessment in search and product discovery depends less on how many models or how much context you use than on how you coordinate them. It proposes HIERA, a three-level hierarchy in which a Relevance Judge decides when specialist input is needed, a Relation Analyzer coordinates a Query Analyzer and an Item Analyzer plus external knowledge, and the Judge makes the final call on the synthesised argument. Across five datasets and eleven baselines, HIERA reports the best accuracy and agreement, with the largest gains on the proprietary EVS set (0.713 vs. 0.517) and consistent double-digit improvements over flat ensembles. The paper's distinctive causal claim is that these gains come from the coordination structure itself: the same agents and the same web knowledge, without hierarchical synthesis, score far lower at nearly the same token budget. If true, this reframes multi-agent evaluation as an architecture-design problem rather than a scaling problem.","feed_headline":"Coordination, not compute, drives LLM relevance gains","feed_subtitle":"Routing specialist analyses through one synthesizer beat 11 baselines and turned extra web knowledge into a 12.7% gain.","key_machinery":"The load-bearing mechanism is the three-layer coordination graph: the Decision Layer's Relevance Judge ($J$) decides whether to delegate; the Analysis Layer's Relation Analyzer ($R$) consults the Query Analyzer ($Q$) and Item Analyzer ($I$) as needed and integrates their findings with Information Layer knowledge ($K$) into one structured relevance argument; the Judge then produces the final label from that argument. $R$ is the central synthesizer that distinguishes the architecture from flat aggregation: specialist outputs pass through $R$ and are interleaved with its own connection analysis before the Judge sees them. The paper defines the Judge's delegation trigger as three evidence dimensions—query interpretability, information completeness, and relational transparency—and reports that the full hierarchy is activated in 91.5% of cases.","core_discovery":"On its own terms, the paper establishes that HIERA outperforms single-step, multi-step, and ensemble LLM relevance assessors on all five evaluation datasets, and that the advantage survives an ablation that fixes the agents and knowledge. The full system reaches 0.859 accuracy on MSRD and 0.681 on WANDS, while the uncoordinated 'Judge + All Analyzers' variant—same specialist outputs, same web search, orchestration-level tokens within 1.8%—reaches only 0.762 and 0.659. The same external knowledge that improves HIERA by 12.7% on MSRD degrades most baselines when concatenated directly into their prompts. From this the authors conclude that the Relation Analyzer's interactive synthesis, not additional computation or context, is the active ingredient, and that coordination topology determines whether adding agents and knowledge helps or hurts.","pith_inferences":["Editorial inference: the same coordination principle should transfer to other LLM families and languages, which the paper does not test; a direct replication with GPT or Gemini models on the public datasets would show whether the hierarchy benefit is Claude-specific.","Editorial inference: because the Judge delegates in 91.5% of cases, HIERA is not acting as an adaptive router; a cheaper variant that predicts delegability and runs the full hierarchy only on hard cases could preserve most of the gain at lower cost, an optimisation the paper leaves open.","Editorial inference: the two documented failure modes suggest targeted fixes—a fact-checking step that prevents specialists from reinforcing an unsupported cultural association, and a softer partial-relevance option when one constraint mismatches but the category matches; both are testable modifications rather than new frameworks.","Editorial inference: if the synthesis effect is truly about information ordering and interactive consultation, a single LLM given the same final structured argument as a chain-of-thought should close part of the gap; that experiment would isolate the benefit of multi-agent routing from the benefit of the reasoning trace itself."],"forward_implications":["Practitioners should expect the largest gains on cases that require connecting separate dimensions, such as implicit color equivalence or style compatibility, where flat and staged baselines systematically underestimate relevance.","Adding web knowledge or extra specialist analyses to an LLM judge can reduce accuracy unless a coordinating layer synthesises them; knowledge integration belongs in the coordination protocol, not appended to prompts.","Multi-agent evaluation comparisons should hold compute and context constant: the paper's topologies differ by only 1.8% in orchestration-level tokens yet differ by 37% in Cohen's $\\kappa$ (0.524 vs. 0.718 on MSRD).","For offline evaluation workloads, the 35-second latency is claimed acceptable because it remains 50 times faster than human experts, so the accuracy gains come at a latency cost that rules out real-time use but suits offline labeling."],"supporting_citations":[{"why":"Establishes the zero-shot LLM relevance judgment baseline and provides the theoretical starting point HIERA extends.","marker":"(Faggioli et al., 2023)"},{"why":"DNA-Prompt single-step baseline showing LLMs can predict searcher preferences; a direct comparison target.","marker":"(Thomas et al., 2024)"},{"why":"JudgeBlender family of ensemble baselines (Prompt-Blender, LLM-Blender) representing flat aggregation without coordination.","marker":"(Rahmani et al., 2025)"},{"why":"CollabEval multi-agent collaborative baseline that broadcasts full histories; the closest peer-based coordination competitor.","marker":"(Qian et al., 2025)"},{"why":"Multi-stage pipeline baseline whose stage isolation HIERA is designed to overcome.","marker":"(Schnabel et al., 2025)"},{"why":"Provides the ESCI e-commerce dataset and its four-level relevance taxonomy used in evaluation.","marker":"(Reddy et al., 2022)"},{"why":"Provides the WANDS furniture-search dataset used for the main ablation and bootstrap intervals.","marker":"(Chen et al., 2022)"},{"why":"ReAct reasoning-and-acting pattern is the implementation backbone for HIERA's tool-based consultations.","marker":"(Yao et al., 2023)"},{"why":"RAG-MMR baseline and the human annotation time estimate (30 minutes per judgment) used for latency comparison.","marker":"(Sachdev et al., 2024)"}],"fun_headline_variants":["HIERA: Coordinated LLM agents deliver 12.7% relevance gain","Hierarchical synthesis, not extra compute, boosts LLM relevance","Why adding LLM agents can hurt without coordination","Same agents, same knowledge: coordination is the difference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the uncoordinated 'Judge + All Analyzers' ablation is a fair control—identical specialist outputs, web knowledge, model versions, prompt templates, and token budgets, differing only by the absence of the Relation Analyzer's synthesis—so the 12.7% accuracy gap can be attributed to coordination rather than to prompt phrasing, output ordering, or unmeasured compute.","fun_headline_variants_meta":{"raw":{"variants":["HIERA: Coordinated LLM agents deliver 12.7% relevance gain","Hierarchical synthesis, not extra compute, boosts LLM relevance","Why adding LLM agents can hurt without coordination","Same agents, same knowledge: coordination is the difference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000644,"raw_usage":{"total_tokens":2955,"prompt_tokens":937,"completion_tokens":2018,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":1948}},"tokens_in":553,"tokens_out":2018,"duration_ms":15390,"temperature":1.0,"reasoning_tokens":1948,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:16:56.324652+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HIERA and the 'Judge + All Analyzers' variant on MSRD with verbatim prompt logs, matched per-agent token limits, the same web-search results, and identical specialist outputs; if the accuracy gap (0.859 vs. 0.762) collapses or fails McNemar's test at $p<0.05$ when prompts and compute are exactly matched, the coordination-structure claim is falsified. A second check is to feed the Judge the exact final argument produced by the Relation Analyzer as plain text in the uncoordinated variant; if that closes most of the gap, the effect comes from the content of synthesis rather than the interactive routing.","supporting_citations":[],"review_version":1}