{"id":"4461dc9a-b365-41a8-838e-c79bb3162fa9","arxiv_id":"2606.11199","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":7,"one_line_summary":"A structured multi-agent RAG pipeline with temporal reranking and bounded contradiction reconciliation won Best Dynamic Evaluation at NeurIPS 2025 by aligning with user preferences over automatic metrics.","lead":"NightFeats is a multi-agent RAG system that decomposes question-answering into retrieval, curation, and composition phases with citation tracking. It won a NeurIPS 2025 competition track for dynamic evaluation, suggesting users prefer transparent, citation-grounded answers over those optimized for surface similarity.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The central comparative claim — that NightFeats surpasses baselines on LLM-as-a-Judge and Human Likert evaluations — is supported by zero quantitative data in the paper. Table 1 shows n/a for both LLM-as-a-Judge columns, no Human Likert scores appear anywhere, and no RAG-Arena win rates or sample si","rationale":"The reader correctly identified the core problem: the central empirical claim is unsupported by data in the paper. I agree with the reader's assessment and would escalate from CONDITIONAL to REJECT. The distinction matters: CONDITIONAL implies the claim would hold if certain gaps were filled; REJECT is warranted when the specific comparative claim made in the abstract and conclusion has no supporting evidence presented whatsoever. Table 1 shows NightFeats scoring lower than both baselines on the only metrics where it has values (ROUGE-L: 0.159 vs 0.181/0.200; BERTScore: 0.840 vs 0.850/0.851), and 'n/a' on the metrics where the paper claims superiority. The paper's own Section 1 states it is 'not a claim of state-of-the-art performance on any single metric,' which contradicts the abstract's assertion. The architectural contributions (structured intermediate representations, temporal-semantic reranking, bounded contradiction reconciliation) are described clearly but without ablation studies showing any component's contribution. As a system description paper, the architecture section has merit. As a paper making a comparative performance claim, it fails to present the evidence needed. If the RAG-Arena data were included with adequate sample sizes and win rates, the verdict could be revisited, but as it stands the central claim is an assertion without visible support.","tokens_in":8107,"tokens_out":1406,"duration_ms":27936,"concrete_test":"Request the official RAG-Arena results from the competition organizers: the number of pairwise comparisons involving NightFeats, the win rate against each baseline (Claude-SonnetV2, Nova-Pro), and any confidence intervals or significance tests. If the win rate is not statistically distinguishable from 50% given the sample size, the 'surpasses' claim does not hold. Additionally, request the Human Likert scores for all systems — these are referenced in the evaluation protocol but absent from Table 1 entirely.","verdict_should_be":"REJECT","load_bearing_attack":"The abstract states NightFeats 'surpasses proprietary baselines including Claude-SonnetV2 and Nova-Pro on LLM-as-a-Judge and Human Likert evaluations.' The conclusion repeats this. But the paper contains no data to support this claim. Table 1 shows 'n/a' for both LLM-as-a-Judge metrics (Sem. Sim. and Fact Cov.) for NightFeats. No Human Likert ratings are reported for any system in the paper. The footnote on Table 1 says LLM-as-a-Judge scores 'are not separately reported in [8]' and that the system was 'evaluated primarily through the live RAG-Arena protocol.' Yet no RAG-Arena win rate, number of pairwise comparisons, sample size, or statistical significance is provided anywhere in the paper. The entire comparative claim rests on the competition award designation ('Best Dynamic Evaluation'), which is an external judgment whose underlying data is not visible to the reader. This is not a case of weak evidence — it is a case of no evidence being presented for the specific comparative claim made. The architectural description (Sections 3–4) is reasonable as a system design, but the empirical claim in the abstract and conclusion is unsupported by anything in the paper itself. The paper even acknowledges in Section 1 that it is 'not a claim of state-of-the-art performance on any single metric,' which directly conflicts with the abstract's assertion of surpassing baselines.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"The paper presents NightFeats, a multi-agent RAG system submitted to the MMU-RAGent NeurIPS 2025 competition. The architecture decomposes knowledge synthesis into three phases—retrieval (ResearchAgent), curation (GeneratorAgent), and composition (WriterAgent)—governed by structured intermediate representations and handoff contracts. Key design primitives include temporal-semantic reranking, bounded contradiction reconciliation (capped at two iterations), and citation-preserving composition. The system was awarded 'Best Dynamic Evaluation' in the text-to-text track.","tokens_in":8968,"tokens_out":1150,"duration_ms":85761,"significance":"The architectural design is clearly articulated and the system addresses genuine limitations of conventional RAG pipelines (provenance drift, retrieval myopia, unbounded iterative drift). The design principles P1–P4 are well-motivated and grounded in prior work (ACE, Reflexion, Self-RAG, CRAG). The temporal-semantic fusion formula (§3.1, Eq. for s_i_tilde) and adaptive cutoff (Eq. for tau) are concrete and reproducible with stated hyperparameters. The bounded curation loop (P3) is a practical contribution. However, the central empirical claim of surpassing baselines is not supported by data within the paper itself.","major_comments":[{"comment":"The abstract states that NightFeats 'surpasses proprietary baselines including Claude-SonnetV2 and Nova-Pro on LLM-as-a-Judge and Human Likert evaluations.' However, Table 1 shows 'n/a' for both LLM-as-a-Judge columns (Sem. Sim. and Fact Cov.) for NightFeats, and no Human Likert scores are reported for any system anywhere in the paper. No RAG-Arena win rates, sample sizes, or statistical significance values are provided. The entire comparative claim rests on the competition award designation, which is an external judgment whose underlying data is not visible to the reader. The abstract and conclusion must be revised to accurately reflect what is and is not demonstrated in the paper, or the supporting quantitative data from the competition must be included.","section":null},{"comment":"Section 1 states: 'This paper presents NightFeats as a system contribution: a reproducible, modular architecture for multi-agent RAG reasoning, not a claim of state-of-the-art performance on any single metric.' This directly conflicts with the abstract's assertion of surpassing baselines on LLM-as-a-Judge and Human Likert evaluations. The framing is internally inconsistent. The authors should reconcile this: either the paper makes a comparative performance claim (requiring data) or it is purely a system description (requiring revised abstract/conclusion language).","section":null},{"comment":"Table 1 shows NightFeats with ROUGE-L=0.159 and BERTScore=0.840, both below the two baselines (Claude-SonnetV2: 0.181/0.850; Nova-Pro: 0.200/0.851). The paper attributes the ROUGE-L gap to citation annotations reducing lexical overlap, which is plausible, but provides no evidence (e.g., ablation with citations removed) to support this explanation. Without any metric where NightFeats outperforms baselines quantitatively, the claim of superiority rests entirely on the undisclosed RAG-Arena results. At minimum, the paper should present whatever RAG-Arena data is available or clearly state that the comparative claim is based solely on the external competition award.","section":null}],"minor_comments":[{"comment":"§3.1: The temporal-semantic reranking formula uses notation s_i_tilde for the fused score but the variable is not typeset as an equation with a number. Consider formatting as a numbered equation to improve cross-referencing.","section":null},{"comment":"§3.1: The chunk size (700 tokens, 100 overlap) and deduplication threshold (0.85 cosine similarity) are stated in prose. Consolidating all hyperparameters (w, alpha, tau_min, chunk_size, chunk_overlap, deduplication_threshold, max_curation_iterations) in a table would improve reproducibility.","section":null},{"comment":"§3.2: The three contradiction severity levels (minor, important, critical) are mentioned but not formally defined. A brief characterization of each level and the corresponding resolution strategy would strengthen the description.","section":null},{"comment":"Figure 1 is referenced but the actual figure content is not visible in the manuscript text. Ensure the figure is properly rendered and that it captures the temporal routing flow described in the text.","section":null},{"comment":"§5: The footnote on Table 1 (†) states LLM-as-a-Judge scores 'are not separately reported in [8].' Clarify whether this means the scores do not exist or are reported only in aggregate form in the competition report.","section":null},{"comment":"References: Several arXiv preprints are cited (e.g., [1], [4]). Where published versions exist, prefer the published reference. Ensure all author names are complete.","section":null}],"recommendation":"major_revision","confidential_remarks":"The core issue is straightforward: the paper makes a comparative performance claim in the abstract and conclusion that is not backed by any data in the paper. Table 1 actually shows NightFeats scoring below both baselines on the two metrics reported. The 'surpasses' claim appears to derive from the RAG-Arena live evaluation, but no win rates or sample sizes are provided. This is fixable if the authors can include the RAG-Arena data or if they revise the abstract/conclusion to accurately scope their claims to what is demonstrated. The architecture description itself is sound and the paper could be a solid system paper with corrected framing."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and fair reading. All three major comments identify a genuine gap between the claims in our abstract/conclusion and the quantitative evidence presented in the paper. We agree with the referee's assessment and will revise accordingly. Below we address each comment in turn.","responses":[{"response":"The referee is correct. The abstract and conclusion overstate what the paper itself demonstrates. Table 1 reports 'n/a' for NightFeats on both LLM-as-a-Judge columns, and no Human Likert scores or RAG-Arena win rates appear anywhere in the manuscript. The claim that NightFeats 'surpasses proprietary baselines on LLM-as-a-Judge and Human Likert evaluations' is not supported by data within the paper. We will revise the abstract and conclusion to accurately state that NightFeats was awarded Best Dynamic Evaluation in the text-to-text track via the RAG-Arena live preference protocol, and that the underlying quantitative data (win rates, sample sizes, significance tests) were not separately reported to us by the competition organizers and are therefore not included in this paper. We will remove the language asserting superiority on LLM-as-a-Judge and Human Likert evaluations unless we can obtain and include the supporting data. We are currently requesting the detailed RAG-Arena results from the competition organizers; if these data become available, we will incorporate them into a revised Table 1 or a supplementary table. If they do not, the revised paper will clearly state that the comparative claim rests solely on the external competition award.","revision_made":"yes","referee_comment":"The abstract states that NightFeats 'surpasses proprietary baselines including Claude-SonnetV2 and Nova-Pro on LLM-as-a-Judge and Human Likert evaluations.' However, Table 1 shows 'n/a' for both LLM-as-a-Judge columns for NightFeats, and no Human Likert scores are reported for any system anywhere in the paper. No RAG-Arena win rates, sample sizes, or statistical significance values are provided. The entire comparative claim rests on the competition award designation, whose underlying data is not visible to the reader."},{"response":"We agree that the framing is internally inconsistent. Section 1 positions the paper as a system description without performance claims, while the abstract makes a comparative superiority claim. This is a drafting error on our part. In the revision, we will reconcile the two: the paper will be consistently framed as a system contribution that documents the architecture as deployed in the MMU-RAGent competition, notes the competition award as an external evaluation outcome, and does not claim quantitative superiority on any metric reported within the paper itself. The abstract will be brought into alignment with the more cautious framing already present in Section 1.","revision_made":"yes","referee_comment":"Section 1 states: 'This paper presents NightFeats as a system contribution: a reproducible, modular architecture for multi-agent RAG reasoning, not a claim of state-of-the-art performance on any single metric.' This directly conflicts with the abstract's assertion of surpassing baselines on LLM-as-a-Judge and Human Likert evaluations. The framing is internally inconsistent."},{"response":"The referee is correct on all counts. NightFeats does not outperform either baseline on any metric reported in Table 1. Our explanation for the ROUGE-L gap—citation annotations reducing lexical overlap—is plausible but currently unsupported by evidence in the paper. We will address this in two ways. First, we will add an ablation experiment in which the WriterAgent's output is post-processed to strip citation annotations before computing ROUGE-L, allowing a direct comparison of citation-annotated versus citation-free outputs under the same retrieval and curation pipeline. This will either confirm or refute our hypothesis. Second, we will revise the results section to state plainly that NightFeats does not outperform baselines on any automatic metric reported in the paper, and that the Best Dynamic Evaluation award reflects the competition's live RAG-Arena preference protocol, whose detailed results are not included in this manuscript. We will add an explicit statement to that effect in both Section 5 and the conclusion. If RAG-Arena data become available from the organizers before revision, we will include them; otherwise, we will not claim quantitative superiority.","revision_made":"yes","referee_comment":"Table 1 shows NightFeats with ROUGE-L=0.159 and BERTScore=0.840, both below the two baselines. The paper attributes the ROUGE-L gap to citation annotations reducing lexical overlap but provides no evidence (e.g., ablation with citations removed). Without any metric where NightFeats outperforms baselines quantitatively, the claim of superiority rests entirely on undisclosed RAG-Arena results. At minimum, the paper should present whatever RAG-Arena data is available or clearly state that the comparative claim is based solely on the external competition award."}],"tokens_in":7935,"tokens_out":1507,"duration_ms":36025,"standing_objections":["The detailed RAG-Arena win rates, sample sizes, and significance values are controlled by the MMU-RAGent competition organizers and were not provided to us in disaggregated form. If the organizers do not release these data, we cannot include them in the paper. In that case, the revised paper will state that the comparative outcome is based solely on the external competition award and that the underlying data are not available to the authors."]},"desk_editor":{"model":"glm-5.2","letter":"The headline: this paper describes a coherent multi-agent RAG architecture but makes a central comparative claim — surpassing Claude-SonnetV2 and Nova-Pro on LLM-as-a-Judge and Human Likert evaluations — that is backed by zero quantitative data in the paper. Table 1 shows n/a for both LLM-as-a-Judge columns, no Human Likert scores appear anywhere, and no RAG-Arena win rates, sample sizes, or significance tests are reported. The claim rests entirely on the competition's award designation, whose underlying data is not visible to the reader. The stress-test concern lands fully here — this is not weak evidence, it is absent evidence for the specific claim being made. The paper even contradicts itself: Section 1 says it is 'not a claim of state-of-the-art performance on any single metric,' while the abstract says it 'surpasses proprietary baselines.' That said, the architectural description is genuinely reasonable work. The three-phase decomposition (retrieval, curation, composition) with structured intermediate representations is clearly articulated. The temporal-semantic reranking formula is simple but principled, with concrete parameter values. The bounded contradiction reconciliation (max 2 iterations, severity classification) is a practical design choice that addresses a real problem. The related work section honestly positions each component as synthesized from prior systems rather than claiming novelty for individual pieces. The seven free parameters (w, alpha, tau_min, chunk_size, etc.) are all specified with values, which is good for reproducibility. The soft spots beyond the missing evidence: the paper has no ablation, no error analysis, and no independent evaluation outside the competition. The latency discussion (10-15s per query) is honest but underspecified. These are minor relative to the central problem. This paper is for practitioners building multi-agent RAG systems who want a reference architecture. The design principles (P1-P4) are useful as engineering guidelines. Whether it deserves a serious referee depends on what the authors are willing to do. If they can obtain and report the actual RAG-Arena win rates, sample sizes, and any Human Likert data from the competition organizers, the paper becomes a solid system contribution. As it stands, the abstract and conclusion make claims the paper cannot support. I'd recommend a serious referee only if the authors commit to either (a) obtaining and reporting the missing competition data, or (b) scaling back the comparative claims to match what Table 1 actually shows.","headline":"Competition system paper with a reasonable architecture but a central empirical claim that is unsupported by any data in the paper itself.","tokens_in":9100,"tokens_out":569,"would_cite":false,"duration_ms":38417,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Citation-First RAG Beats Proprietary LLMs in Live User Preferences","keywords":["retrieval-augmented generation","multi-agent systems","citation traceability","temporal reranking","contradiction reconciliation","human preference evaluation","RAG-Arena","NeurIPS 2025"],"falsifier":"A controlled study with reported sample sizes and statistical tests showing that NightFeats does not significantly outperform proprietary baselines on human preference or LLM-as-a-Judge evaluations when queried on identical question sets outside the competition environment. Additionally, if the temporal-semantic reranking weight (w=0.3) were ablated and shown to have no measurable effect on user preference, or if the bounded contradiction reconciliation loop (capped at 2 iterations) were shown to produce no improvement in factual coherence over a single-pass pipeline, the core architectural","tokens_in":8380,"feed_emoji":"🔍","tokens_out":961,"duration_ms":99521,"temperature":0.7,"pith_summary":"This paper presents NightFeats, a multi-agent retrieval-augmented generation system that decomposes knowledge synthesis into three phases—retrieval, curation, and composition—each governed by explicit intermediate data models and handoff contracts between agents. The central claim is that architectural transparency, verifiable evidence grounding, and citation traceability are better aligned with real user preferences than systems that optimize for automatic similarity metrics like ROUGE-L or BERTScore. The system introduces three core mechanisms: a temporal-semantic reranking formula that fuses semantic relevance with document recency, a bounded contradiction reconciliation loop that caps iterative refinement at two sub-retrieval cycles to prevent recursive drift, and citation-preserving composition that enforces source traceability as a structural invariant rather than a generation artifact. At the NeurIPS 2025 MMU-RAGent competition, NightFeats was awarded Best Dynamic Evaluation in the text-to-text track via live pairwise user preferences on the RAG-Arena platform, where it was preferred over proprietary baselines including Claude-SonnetV2 and Nova-Pro. The authors argue this outcome validates their design thesis: users value factual traceability and temporal currency even when standard automatic metrics penalize citation-annotated outputs for reduced lexical overlap with reference answers.","feed_headline":"Citation-First RAG Beats Proprietary LLMs in Live User Votes","feed_subtitle":"A three-agent pipeline with built-in citation traceability and temporal reranking outperforms Claude and Nova-Pro on real user preferences, ","key_machinery":"The system coordinates three asynchronously operating agents built on the OpenAI Agents SDK with GPT-5. The ResearchAgent executes a seven-step retrieval pipeline: query expansion (1-3 subqueries), retrieval via Jina Reader over Brave Search (Recent mode) or FineWeb corpus (Historical mode), semantic chunking (700 tokens, 100 overlap), embedding with text-embedding-3-small, temporal-emantic reranking via Voyage AI using the fusion formula s_i_tilde = (1-w)*s_i + w*1/(1+d_i/365) with w=0.3, adaptive cutoff using tau = max(tau_min, mu - alpha*sigma) with alpha=1.0 and tau_min=0.2, and deduplication pruning above 0.85 cosine similarity. The GeneratorAgent extracts atomic cited facts, clusters","core_discovery":"The paper's central contribution is a three-agent RAG pipeline (ResearchAgent, GeneratorAgent, WriterAgent) where each stage outputs a standardized intermediate representation carrying provenance metadata, contradiction annotations, and normalized citations. The key architectural primitives are: (1) a temporal-semantic fusion score that combines a semantic reranker score with a recency decay term weighted at w=0.3, discounting stale but topically similar documents; (2) a bounded contradiction reconciliation loop that classifies detected contradictions by severity and triggers at most two targeted sub-retrieval cycles, achieving factual coherence without unbounded iterative drift; and (3) a c","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Three-Agent RAG Beats Claude and Nova-Pro on Human Preferences","Bounded Contradiction Loop Keeps Multi-Agent RAG Grounded","Temporal-Semantic Reranking Stabilizes RAG Output Quality","Structured Handoffs Beat Benchmark-Tuned RAG in User Votes","Citation-First RAG Outperforms Proprietary Baselines on Likert"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The claim that NightFeats surpasses proprietary baselines on human preference evaluations rests on the RAG-Arena live pairwise comparison results, but the paper provides no quantitative data on win rates, sample sizes, or statistical significance for that evaluation, and Table 1 shows 'n/a' for NightFeats's LLM-as-a-Judge scores. The weakest assumption is that the competition's dynamic evaluation award constitutes sufficient evidence to confirm the general superiority of this","fun_headline_variants_meta":{"raw":{"variants":["Three-Agent RAG Beats Claude and Nova-Pro on Human Preferences","Bounded Contradiction Loop Keeps Multi-Agent RAG Grounded","Temporal-Semantic Reranking Stabilizes RAG Output Quality","Structured Handoffs Beat Benchmark-Tuned RAG in User Votes","Citation-First RAG Outperforms Proprietary Baselines on Likert","ACE-Inspired Pipeline Decomposes RAG Into Retrieval, Curation, Composition","Verifiable Provenance Metadata Drives RAG Human Preference Win","Two-Cycle Contradiction Reconciliation Prevents RAG Drift","Three-Phase RAG Beats Proprietary LLMs Without Metric Gaming","Intermediate Representations With Citation Metadata Improve RAG Trust"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":724,"prompt_tokens":500,"completion_tokens":224,"prompt_tokens_details":null},"tokens_in":500,"tokens_out":224,"duration_ms":15531,"temperature":1.0,"reasoning_tokens":65,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-05T07:03:50.694371+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"A controlled study with reported sample sizes and statistical tests showing that NightFeats does not significantly outperform proprietary baselines on human preference or LLM-as-a-Judge evaluations when queried on identical question sets outside the competition environment. Additionally, if the temporal-semantic reranking weight (w=0.3) were ablated and shown to have no measurable effect on user preference, or if the bounded contradiction reconciliation loop (capped at 2 iterations) were shown to produce no improvement in factual coherence over a single-pass pipeline, the core architectural","supporting_citations":[],"review_version":1}