{"id":"7b7b16d0-0cf6-458c-9cae-8e64d27cf196","arxiv_id":"2501.13946","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A three-agent pipeline with OVON JSON messages lowers the authors' disclaimer-based hallucination score on 310 prompts, but that score does not measure truth.","lead":"The authors built a three-agent pipeline in which a second and third AI rewrite a first AI's answers to 310 tricky prompts, passing notes through a JSON format called OVON. They report a drop in their new 'hallucination score' at each stage, but the score mostly rewards the agents for adding disclaimers, so the drop may show instruction-following rather than real factual improvement.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"THS is constructed to reward disclaimers (FDF/ECS enter negatively) while the front-end agent is explicitly forbidden from adding them; the reported drop is an instruction-following artifact unless a control shows otherwise.","rationale":"The strongest claim—that multi-agent orchestration mitigates hallucinations—requires that the outcome measure used in Table 2 is a valid proxy for hallucination/factuality. The paper's own text undermines this: Section 4 states the KPIs 'don't rely only on absolute factual truth but rather on linguistic and stylistic indicators,' and Section 8 calls them 'not an infallible benchmark for factual correctness.' More importantly, THS is not just a noisy proxy; it is directionally biased toward the intervention. The front-end prompt forbids disclaimers, the reviewer prompts require them, and FDF+ECS enter the THS formula with negative signs. Thus the reported monotonic decrease is expected even if every fabricated claim survives unchanged. This is a confound rather than a mere measurement limitation. A cheap deterministic control—appending a scripted disclaimer to the 310 front-end outputs—would determine whether the effect is an artifact of the metric. The reader's weakest assumption identified the same issue (THS validity), so I agree with their assessment. The public repository and 310-prompt dataset are real assets, but reproducibility of the computation does not establish validity of the metric. Because the provided evidence does not support the central claim, the REJECT verdict is appropriate; my analysis does not move it.","tokens_in":16295,"tokens_out":4767,"duration_ms":53352,"concrete_test":"Take the 310 front-end outputs and apply a deterministic post-processor that appends one sentence explicitly labeling the content as speculative/fictional (e.g., 'This description is purely fictional and has no verified factual basis'), leaving all factual content unchanged; score both original and post-processed outputs with the same Section 4 THS formula and the same GPT-4o KPI evaluator. If the mean THS drops by an amount comparable to the reported THS2-THS1 or THS3-THS1 deltas (-0.041 and -0.135), the headline result is explained by surface disclaimer insertion rather than by the multi-agent OVON pipeline, and the central hallucination-mitigation claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that multi-agent OVON processing mitigates hallucinations rests entirely on the Total Hallucination Score defined in Section 4: THS = FCD - (FGR+FDF+ECS)/NA. Since FDF and ECS appear with negative signs, any response that adds fictional disclaimers or explicit contextualization statements lowers THS regardless of whether the underlying factual errors, fabrications, or unsupported claims are corrected. The Section 5 agent prompts guarantee this confound: the front-end agent is told 'Do not ever add any explicit disclaimer', while the second- and third-level agents are instructed to 'Explicit disclaimers wherever content is speculative or fictional'. Thus the mean THS progression -0.0049 -> -0.0456 -> -0.1396 in Table 2 is largely a measure of instruction following, not of hallucination mitigation. The paper itself concedes in Section 8 that the KPIs 'do not offer an infallible benchmark for factual correctness' and in Section 4 that they 'don't rely only on absolute factual truth'. No independent factuality baseline, no external fact-checking metric, and no single-agent control is reported. Without such a control, the observed THS reduction cannot support the paper's strongest claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-agent pipeline for LLM hallucination mitigation: a front-end GPT-3.5-turbo agent answers 310 prompts designed to induce hallucinations; a second-level GPT-4o reviewer rewrites with explicit disclaimers and generates OVON JSON utterances and whispers; a third-level GPT-4o reviewer further refines the text; and a fourth GPT-4o agent computes four new KPIs (Factual Claim Density, Factual Grounding References, Fictional Disclaimer Frequency, Explicit Contextualization Score) and a Total Hallucination Score (THS). The authors report that the mean THS decreases from -0.0049 to -0.0456 to -0.1396 across the three agent stages and interpret this as evidence that multi-agent orchestration, especially with OVON-based JSON exchanges, mitigates hallucinations.","tokens_in":16500,"tokens_out":5242,"duration_ms":53902,"significance":"If the THS were a valid measure of hallucination and the experimental design included appropriate controls, the paper would be a useful practical demonstration of an open-standard NLP-based agent interoperability framework for hallucination mitigation, with a welcome public repository of prompts and results. The proposed OVON JSON message exchange is a concrete contribution to agentic AI interoperability. However, the claimed empirical evidence is not credible as presented: the outcome metric is defined to reward the very behaviors the agents are explicitly instructed to produce, and the manuscript's own limitations section concedes that the KPIs are not a benchmark of factual correctness. The framework idea is interesting, but the central claim that the pipeline mitigates hallucinations is not supported by the reported experiments.","major_comments":[{"comment":"The THS formula includes FDF and ECS with negative signs, so any increase in explicit disclaimers or contextualization statements lowers THS by construction. Section 5 instructs the second- and third-level agents to \"Explicit disclaimers wherever content is speculative or fictional, even if it is a repetition,\" while the front-end agent is told \"Do not ever add any explicit disclaimer.\" Thus the monotone decrease in mean THS in Table 2 (-0.0049, -0.0456, -0.1396) is a direct consequence of the agents following those instructions. No control condition (e.g., a rewriting agent without disclaimer instructions) and no external factuality measure (e.g., known-answer prompts, human fact-checking, or an established factuality benchmark) is reported, so the observed THS reduction does not demonstrate hallucination mitigation.","section":"§4, Eq. (1); §5, agent prompts; Table 2"},{"comment":"The limitations section states that the KPIs \"do not offer an infallible benchmark for factual correctness,\" and Section 4 says they \"don't rely only on absolute factual truth.\" Since the central claim of the paper is hallucination mitigation—a factual-correctness phenomenon—the proposed metric cannot bear that claim by itself. The paper needs an external criterion linking THS reductions to real reductions in factually wrong or fabricated content. Without such validation, the results measure surface-level lexical framing rather than hallucination.","section":"§8, Limitations; §4, Hallucination KPIs"},{"comment":"The paper claims the results are \"statistically significant, as shown by a one-way Analysis of Variance,\" but no test statistic, p-value, or effect size is reported, making the claim unverifiable. In addition, Section 7 reports \"THS1 of -1.52,\" \"THS2 plummets to -14.12,\" and \"THS3 to -43.27,\" which are inconsistent with the per-prompt means in Table 2 unless they are cumulative sums over 310 prompts, which is not stated. The \"percentage reductions surpassing 800%\" and \"nearly 2,800%\" are computed from negative scores and are misleading without careful definition of the baseline.","section":"§5, Empirical Results; §7, Result discussion"},{"comment":"The use-case example concretely illustrates the evaluation artifact: the Library of Avencord response shows THS decreasing from -0.0083 to -0.0667 while FCD changes only from 0.2 to 0.1 and FGR actually increases from 0.1 to 0.2. The entire score reduction is driven by FDF (0.1 to 0.3) and ECS (0.1 to 0.4)—exactly the disclaimer-related metrics that the second- and third-level agents are prompted to increase. This example shows that the reported improvement is an instruction-following effect rather than evidence of correcting fabricated content.","section":"§6, Use case"}],"minor_comments":[{"comment":"The titles of references [8] and [9] contain a typo: \"hallucinatoin\" should be \"hallucination.\"","section":"References [8], [9]"},{"comment":"The citation numbering is inconsistent: Section 1 cites Automated Design of Agentic Systems (ADAS) as [21], but Section 9 cites the same concept as [13], and reference [21] is the \"Hallucination is Inevitable\" paper. Please re-map the references properly.","section":"References [21], [13]"},{"comment":"The symbol NA is used to denote the number of agents, but NA is easily read as \"not available.\" Use a different notation, such as N_agents, and define it before the equation.","section":"§4, Eq. (1)"},{"comment":"The text describes spikes, dips, and prompt IDs in Figures 2 and 3, but the figures are not accompanied by explicit axis labels or units. Ensure that figure files include labeled axes and that the y-axis is identified as THS or Delta THS with clear units.","section":"Figures 2 and 3"},{"comment":"The statements \"surpassing 800%\" and \"nearly 2,800%\" reductions should be re-expressed in a way that is not misleading for negative-valued scores, or the baseline should be explicitly defined.","section":"§7, Result discussion"}],"recommendation":"reject","confidential_remarks":"The paper has a useful public artifact and showcases the OVON interoperability framework, but the empirical evaluation is circular: the outcome metric is constructed to decrease when agents follow their explicit instructions to add disclaimers, and the authors' own limitations section concedes that the KPIs do not measure factual correctness. This is a load-bearing flaw in the central claim, not a presentation issue. I would encourage the authors to add a human-annotated factuality benchmark, a single-model control without disclaimer instructions, and a sensitivity analysis of the KPI weights, and to resubmit as a substantially revised evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a concrete demonstration of an OVON-based multi-agent pipeline that inserts disclaimers into speculative LLM output. That part works. But the headline claim—that this mitigates hallucinations—doesn't survive contact with the evaluation. The Total Hallucination Score is built to reward exactly what the second and third agents are instructed to do: add explicit disclaimers and reduce factual claim density. The front-end agent is told never to add disclaimers. So the observed THS drop from -0.0049 to -0.1396 is mostly a measure of instruction following, not a measure of factual accuracy. The paper itself admits in Section 8 that the KPIs 'do not offer an infallible benchmark for factual correctness.'\n\nWhat's genuinely useful: the OVON whisper messages are a reasonable way to pass hallucination-risk metadata between agents, and the authors ship the code, prompts, and responses in a public repo. The four KPIs are clearly defined and operationalized by an LLM evaluator. Even if they don't measure truth, they measure transparency about uncertainty, which is a legitimate (if narrower) property.\n\nThe soft spots are serious. No baseline without disclaimers, no external factuality check, no single-agent control, and the reported ANOVA lacks the actual F/p values—just a claim of significance. The weights in the THS formula are hand-set to 0.25 each without sensitivity analysis. The discussion in Section 7 inflates small numeric shifts into '800% reductions,' which is misleading when the baseline is near zero. None of this is fatal to the idea that structured inter-agent messaging can change output style, but it's fatal to the paper's central claim about hallucination mitigation.\n\nWho is this for? Someone working on agent interoperability or on designing evaluation metrics for transparency. They'll get a useful example of how not to validate a mitigation method. A serious referee could push the authors toward a factual benchmark (e.g., claim verification or human fact-checking) and a control condition, and the paper could become a solid negative-methodology lesson. I'd send it to review rather than desk reject, because there is reproducible engineering here and a field that needs to see this failure mode clearly.","headline":"A useful exhibition of OVON inter-agent messaging, but the THS metric rewards disclaimer insertion rather than factuality, so the central hallucination-mitigation claim is not established.","tokens_in":17050,"tokens_out":3234,"would_cite":false,"duration_ms":31112,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Routing LLM outputs through two successive reviewer agents that pass structured natural-language notes measurably reduces a composite hallucination score across 310 crafted prompts.","keywords":["hallucination mitigation","multi-agent orchestration","agentic AI","large language models","OVON interoperability","natural language APIs","hallucination KPIs","AI explainability"],"falsifier":"Run the same 310 prompts through the pipeline and have independent human fact-checkers score the front-end and third-level outputs for factual accuracy and disclaimer adequacy; if the more negative THS3 outputs are not also judged more accurate or more clearly labeled as speculative, the claimed mitigation is an artifact of the scoring metric. A sharper test: take third-level outputs, delete all disclaimers without changing any facts, and recompute the THS—the score would rise even though factuality is unchanged.","tokens_in":16052,"feed_emoji":"🤖","tokens_out":9209,"duration_ms":91957,"temperature":0.7,"pith_summary":"The paper claims that passing an LLM's initial answer through two successive reviewer agents—each instructed to flag unverified claims and insert explicit disclaimers—progressively lowers a composite hallucination score. On 310 prompts deliberately built to elicit fabrication, the mean Total Hallucination Score moves from $-0.0049$ for the front-end agent to $-0.0456$ after the second-level reviewer to $-0.1396$ after the third-level reviewer, with the authors reporting this trend as statistically significant. The intended payoff is practical: hallucination mitigation does not require replacing the generator; it can be achieved by orchestration, provided the agents hand off context through structured natural-language messages. The paper also proposes four KPIs that quantify how 'fact-sounding' and how explicitly disclaimed a response is, so that shifts in hallucination behavior can be measured and audited.","feed_headline":"Hallucination scores drop at each agent stage across 310 prompts","feed_subtitle":"Iterative review with structured natural-language handoffs gives a practical recipe for flagging speculative AI output.","key_machinery":"The load-bearing object is the OVON 'conversation envelope,' a JSON format in which the utterance event carries rewritten text and the whisper event carries a natural-language explanation of hallucination likelihood (a context summary capped at 30 words and a reasons field capped at 200 words). The second-level reviewer emits this envelope and the third-level reviewer interprets it, which is what lets the refinement build on prior analysis instead of starting fresh. The four KPIs are Factual Claim Density (fact-sounding claims per 100 words), Factual Grounding References (explicit appeals to real-world evidence), Fictional Disclaimer Frequency (explicit fiction markers per 100 words), and Explicit Contextualization Score (normalized count of overt fictional-framing phrases). They are combined into the composite score $\\mathrm{THS} = \\mathrm{FCD} - (\\mathrm{FGR}+\\mathrm{FDF}+\\mathrm{ECS})/N_A$, with all four weights set to 0.25 in this experiment; because more disclaimers and less fact-sounding phrasing push the value more negative, the pipeline's success is registered by a single number that falls at each stage.","core_discovery":"The central discovery is that structured multi-agent review measurably reduces the paper's Total Hallucination Score across a large, deliberately adversarial prompt set. The front-end agent produces fluent but often ungrounded narrative; the second-level reviewer rewrites the text, adds disclaimers, and attaches a short 'whisper context' plus a detailed 'whisper value' describing the hallucination risk; the third-level reviewer consumes that JSON envelope and produces a final, more explicitly framed version. The authors attribute the monotone improvement—both in mean scores and in aggregate totals, from $-1.52$ to $-14.12$ to $-43.27$—to this iterative layering and to the OVON message handoff that preserves the second agent's reasoning. A direct corollary they draw is that content becomes more transparent, with speculative material clearly demarcated from factual claims, even when the underlying facts remain unverifiable.","pith_inferences":["Editorial extension: because the THS rewards disclaimers and penalizes fact-sounding phrasing, the pipeline may be lowering the appearance of hallucination as much or more than the factual error rate; an independent human factuality benchmark would separate the two effects.","A direct ablation test follows from the paper's design: run the same three-agent pipeline but send only the plain rewritten text to the third agent, omitting the OVON whisper fields; any THS gap would isolate the contribution of structured context transfer from the contribution of a second generic rewrite.","The same four KPIs could be reused outside hallucination research as a transparency score—for example, to measure how much of any AI answer is presented as fact versus explicitly labeled speculation.","Because the front-end model is older and weaker than the reviewers, part of the improvement may reflect model capability differences rather than the agentic orchestration; a same-model baseline would test that confound."],"forward_implications":["If the result holds, a concrete recipe exists for lowering hallucination scores: add a rewriting agent that inserts disclaimers, add a second refining agent, and pass a short structured explanation of hallucination risk between them.","The OVON JSON handoff can be extended to all inter-agent exchanges, not just the second-to-third transition, which the paper explicitly leaves as future work.","The wide prompt-to-prompt variance—a roughly 700% score reduction for the Library of Avencord prompt versus 33% for the telepathic-canines prompt—implies the method works best when speculative content can be anchored to familiar real-world concepts.","The KPI formulas provide an automated, auditable target for comparing reviewer prompts, agent models, and pipeline depths without human annotation of every response."],"supporting_citations":[{"why":"Defines the OVON universal NLP-based API that the paper uses as the inter-agent communication standard.","marker":"[10]"},{"why":"Specifies the Conversation Envelope JSON format with utterance and whisper events that carry text and hallucination metadata between agents.","marker":"[1]"},{"why":"Supplies the public dataset of 310 prompts and agent responses from which the reported THS values are computed.","marker":"[8]"},{"why":"Earlier multi-agentic hallucination-mitigation study whose detection and revision results motivate the review-pipeline design.","marker":"[15]"},{"why":"Multi-agent debate approach that the paper contrasts with its sequential review architecture.","marker":"[16]"},{"why":"Argues hallucinations cannot be eliminated in LLMs, framing mitigation by orchestration as the realistic goal.","marker":"[21]"},{"why":"Provides the check-worthy claim-detection concept underlying the Factual Claim Density KPI.","marker":"[12]"},{"why":"Measures attribution in generation models and grounds the Factual Grounding References KPI.","marker":"[18]"}],"fun_headline_variants":["Agentic NLP framework reduces AI hallucinations via layered review","Multi-agent JSON handoffs cut hallucination scores in generative AI","Three-stage AI review drops hallucination scores on 310 prompts","Structured agent pipeline mitigates AI hallucinations with measurable KPIs","Layered AI agents use OVON handoffs to lower hallucination metrics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the KPI-based Total Hallucination Score, which is computed by an LLM evaluator from surface features like how many claims sound factual and how often disclaimers appear, actually tracks hallucination or factuality—a point the paper itself states when it says these KPIs do not rely on absolute factual truth and are not an infallible benchmark.","fun_headline_variants_meta":{"raw":{"variants":["Agentic NLP framework reduces AI hallucinations via layered review","Multi-agent JSON handoffs cut hallucination scores in generative AI","Three-stage AI review drops hallucination scores on 310 prompts","Structured agent pipeline mitigates AI hallucinations with measurable KPIs","Layered AI agents use OVON handoffs to lower hallucination metrics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1548,"prompt_tokens":992,"completion_tokens":556,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":469}},"tokens_in":608,"tokens_out":556,"duration_ms":5863,"temperature":1.0,"reasoning_tokens":469,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:42:50.540311+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 310 prompts through the pipeline and have independent human fact-checkers score the front-end and third-level outputs for factual accuracy and disclaimer adequacy; if the more negative THS3 outputs are not also judged more accurate or more clearly labeled as speculative, the claimed mitigation is an artifact of the scoring metric. A sharper test: take third-level outputs, delete all disclaimers without changing any facts, and recompute the THS—the score would rise even though factuality is unchanged.","supporting_citations":[{"cited_title":"Conversational AI Multi-Agent Interoperability, Universal Open APIs for Agentic Natural Language Multimodal Communications","cited_arxiv_id":"2407.19438","evidence_quote":"Defines the OVON universal NLP-based API that the paper uses as the inter-agent communication standard."},{"cited_title":"Open voice interoper- ability dialog conversation envelop specifications","cited_arxiv_id":null,"evidence_quote":"Specifies the Conversation Envelope JSON format with utterance and whisper events that carry text and hallucination metadata between agents."},{"cited_title":"Multi agent agentic hallucinatoin evaluator","cited_arxiv_id":null,"evidence_quote":"Supplies the public dataset of 310 prompts and agent responses from which the reported THS values are computed."},{"cited_title":"Good Parenting is all you need -- Multi-agentic LLM Hallucination Mitigation","cited_arxiv_id":"2410.14262","evidence_quote":"Earlier multi-agentic hallucination-mitigation study whose detection and revision results motivate the review-pipeline design."},{"cited_title":"Toward automated fact-checking: Detecting check-worthy factual claims by claimbuster","cited_arxiv_id":null,"evidence_quote":"Provides the check-worthy claim-detection concept underlying the Factual Claim Density KPI."}],"review_version":1}