{"id":"6b8009ae-c5fe-4bd6-90f1-b71b9f9c611a","arxiv_id":"2412.05223","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Acurai reports 100% hallucination-free outputs on 37 RAGTruth conflict examples for GPT-4 and GPT-3.5 Turbo by rewriting queries and context into simplified 'Fully-Formatted Facts' and splitting similar terms.","lead":"Acurai, Inc. claims a systematic input-rewriting method that eliminates 100% of hallucinations in GPT-4 and GPT-3.5 Turbo on the RAGTruth corpus. The evidence is a self-scored evaluation of 37 conflict examples, with no check on whether rewriting introduces new errors on correct answers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 100% claim is supported only by an unblinded, author-run evaluation on 37 conflict cases; no test on non-conflict RAGTruth examples or other hallucination types is reported, so the title-level claim is unestablished.","rationale":"The title makes a categorical claim: 100% elimination of hallucinations on RAGTruth for GPT-4 and GPT-3.5 Turbo. To establish that, one needs an unbiased measurement and a population that matches the claim. Neither condition is met. The 37 conflict examples are a tight, favorable sample: every base response is already a hallucination, the passages are relevant and small in number, and the authors performed the faithfulness comparison themselves after discarding RAGTruth's own labels. A 100% rate on such a sample is exactly what one would expect if the transformation simply turns each query into several subqueries and sends only hand-picked relevant sentences; the model is then being asked to summarize a near-answer rather than reason over conflicting or irrelevant context. The paper's own limitations section narrows the setting further, but the title and abstract do not carry those qualifications. The most decisive check is to expand the evaluation to the full RAGTruth corpus and to blind independent annotators; if Acurai still scores 100% on all GPT-4 and GPT-3.5 Turbo instances, the concern about sample selection is resolved, and if not, the title-level claim fails. This is a concrete, practical check that speaks directly to the central claim. The issue is not that the authors are dishonest; it is that the evidence as presented cannot support the categorical wording of the claim.","tokens_in":9077,"tokens_out":6214,"duration_ms":66332,"concrete_test":"Apply the Acurai pipeline to all RAGTruth query-passage-response triples for GPT-4 and GPT-3.5 Turbo, not only the 37 conflict cases, including previously correct and non-conflict responses. Have two annotators who are blind to which outputs are Acurai-processed and to the study hypothesis independently compare each response to the original unmodified passages, using pre-registered faithfulness criteria and reporting inter-annotator agreement. If any Acurai output deviates from the source passages, or if the annotators disagree on any of the 37 original cases, the claimed 100% elimination is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that Acurai eliminates 100% of hallucinations on RAGTruth for GPT-4 and GPT-3.5 Turbo. The evidence in Section 4.2 is a manual comparison by the authors of 37 Acurai outputs against the original passages, after discarding RAGTruth's own annotations. This is not an independent measurement, and it covers only the four conflict categories: GPT-4 and GPT-3.5 Turbo, each with Subtle and Evident Conflict. Three load-bearing gaps follow. First, the evaluated population is restricted to cases already known to produce hallucinations; there is no check that Acurai's rewriting does not introduce hallucinations on previously correct outputs or on other RAGTruth hallucination categories, such as unverifiable content. Second, the transformation itself is not fully specified: the paper gives examples of splitting queries and hand-curating 'Fully-Formatted Facts' in Section 3.2, but no algorithm, prompts, or thresholds are provided, so the '100%' result cannot be reproduced or audited from the text. Third, the comparison is unblinded and author-conducted, so leniency in judging whether a response 'deviated from the provided passages in any manner whatsoever' is a live risk. The confidence interval [0.91, 1] is correctly calculated but applies only to the 37 cases; it does not license the title's unconditional '100% Elimination on RAGTruth.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Acurai, a pre-processing pipeline that rewrites queries and context passages to avoid what the authors call 'noun-phrase collisions,' and claims that this method eliminates 100% of hallucinations in the RAGTruth corpus for GPT-4 and GPT-3.5 Turbo. The evaluation in Section 4.2 compares Acurai-transformed responses to the original passages for 37 conflict examples across four categories (GPT-4/GPT-3.5 Turbo × Subtle/Evident Conflict), reporting a perfect success rate and a Wilson confidence interval of [0.91, 1]. The paper also presents a theoretical model asserting that LLMs self-organize around noun phrases, supported by self-citations to blog posts and videos.","tokens_in":9279,"tokens_out":4108,"duration_ms":37321,"significance":"If the 100% claim were established, Acurai would be a notable advance for RAG faithfulness, with potentially wide application in enterprise settings. The paper does provide a concrete dataset and full responses on GitHub, which is a useful starting point for reproducibility. However, the current evidence falls far short of the title-level claim: the evaluation is small, self-scored, restricted to known conflict cases, and the method itself is not specified in sufficient detail to be reproduced. The theoretical framework is not empirically validated. As presented, the paper is better read as a proposal or company announcement than as a demonstrated scientific result.","major_comments":[{"comment":"The 100% claim is evaluated only on 37 conflict examples (the four RAGTruth conflict categories for GPT-4 and GPT-3.5 Turbo). The abstract and title claim 100% elimination of hallucinations on RAGTruth, but RAGTruth contains multiple hallucination categories (e.g., unverifiable content) and non-conflict cases; no evaluation is reported on those. A perfect score on a convenience sample of cases already known to produce hallucinations does not support the general 'on RAGTruth' claim.","section":"Section 4.2 / Title"},{"comment":"The evaluation metric is defined and scored by the authors: responses are compared against the original passages for any deviation 'in any manner whatsoever,' but no operational criteria or rubric are provided, and the comparison is unblinded. Section 4.1 discards the RAGTruth annotator notes for three cited defects, but instead of correcting specific annotations, the paper discards all external human judgments. The risk of leniency in self-scoring is material and undermines the '100%' result as an independent measurement.","section":"Section 4.2"},{"comment":"The method is described through examples, not an algorithm. Step One gives a heuristic to split queries on noun-phrase collisions, but no rule is given for identifying collisions or determining the number of splits. Step Two says passages are rewritten into 'Fully-Formatted Facts' but provides no template, no prompt, and no criteria for what counts as an FFF. Without these specifications, the reported 100% result is not reproducible from the paper, and the GitHub repo (which contains responses) does not appear to include the transformation code or intermediate prompts.","section":"Section 3"},{"comment":"The Wilson interval is correctly computed for n=37, but the interpretation that 'Acurai is expected to eliminate between 91% to 100% of hallucinations in the general population under the same conditions' is unjustified. The sample is not a random sample from a defined population; it is the subset of RAGTruth cases already known to produce hallucinations. The confidence interval therefore applies only to the 37 tested cases, not to RAGTruth as a whole or to any broader population.","section":"Section 4.3"},{"comment":"The Noun-Phrase Dominance Model is asserted as the theoretical basis, but the only supporting evidence is self-citations to a blog and YouTube videos (refs 13, 15, 16, 17). The central axiom that 'all hallucinations occur when the LLM mistakes two distinct Noun-Phrases as being the same thing' is a strong universal claim with no empirical evidence in this paper, and it is not established by the cited studies on LLM features. This matters because the entire Acurai pipeline is motivated by this model; if the model is wrong, the method lacks a rationale.","section":"Section 2"}],"minor_comments":[{"comment":"The statement that 'no LLM, regardless of the amount of data sent to it, was able to exceed greater than 80% accuracy' is too strong given that the cited reference is a single benchmark study; the claim should be qualified to the specific models and tasks tested.","section":"Section 1"},{"comment":"The term 'noun-phrase collisions' is used without a formal definition; in particular, 'chemical properties & physical properties' are called a collision pair, but their relation to noun-phrase identity is not explained.","section":"Section 3.1"},{"comment":"The examples of 'missing instructions' and 'wrong information' are anecdotal; the paper does not state how many RAGTruth entries were discarded or re-annotated, making it impossible to quantify the bias from discarding annotator notes.","section":"Section 4.1"},{"comment":"The GitHub URL is provided, but no version or commit identifier is given, making it difficult to refer to the exact dataset and responses used for the evaluation.","section":"Section 4.2"},{"comment":"References [6], [15], [16], and [17] are blog posts or YouTube videos; these are not peer-reviewed sources and should be supplemented with archival work, especially since they are used to ground the theoretical claims.","section":"References"}],"recommendation":"reject","confidential_remarks":"The commercial affiliation of the authors (Acurai, Inc.) is not in itself a problem, but the paper reads as a product announcement rather than a scientific study. The central claim is unsupported by the presented evidence, the method is underspecified, and the evaluation does not meet the standards of a peer-reviewed journal. A rejection is appropriate; should the authors later provide a proper independent evaluation with a fully specified algorithm, a revised submission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear X,\n\nThe headline is simple: this paper claims 100% elimination of RAG hallucinations, but the evidence is a 37-example, unblinded, author-run eval on cases already known to fail. The claim doesn't hold. That said, there is a real, if modest, nugget here: a concrete recipe for rewriting queries to separate confusable entities and rewriting context into short factual statements, with placeholders for tricky tokens. They publish their responses on GitHub, which is more than many papers do, and they include a Limitations section that concedes scope limits (factual passages, few passages, one model family, extra cost/latency). The Wilson interval is correctly computed; it just doesn't license the title.\n\nThe soft spots are exactly where the stress-test lands. First, the evaluation compares Acurai outputs to the passages themselves without any independent annotation, and on 37 examples that were all selected because the base model had hallucinated. There is no check on non-conflict examples, no check for new hallucinations produced by rewriting correct answers, no baseline, and no per-category counts. Second, the method isn't specified enough to reproduce: the paper gives illustrative examples of splitting and 'Fully-Formatted Facts', but no algorithm, prompts, or thresholds. Third, the theoretical motivation (Noun-Phrase Dominance Model, DFUs) is asserted with self-citations to videos and a blog; DFUs are never defined in the abstract or body. Fourth, the paper discards the RAGTruth annotations and substitutes its own judgment, which is fine as a critique of the dataset but undermines comparison to any prior numbers.\n\nThe limitations section is honest about scope, but it omits the three issues that actually matter: the tiny conflict-only sample, the self-scoring, and the lack of a reproducibility specification. If the authors had treated this as a small case study and framed it as 'we show rewriting can fix these 37 cases', I'd have more sympathy. As published, the abstract overclaims.\n\nWho should read it? Practitioners interested in prompt-rewriting heuristics might get ideas. As a paper, it is not ready for a serious venue. I would desk-reject it; if the stakes were lower, it could be a workshop demo. Do not cite it as evidence.\n\nBest,","headline":"The 100% claim is unsupported by the evidence: 37 self-scored conflict examples, no baseline, and no reproducible specification.","tokens_in":9893,"tokens_out":3201,"would_cite":false,"duration_ms":33870,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reformatting queries and passages into simple fact statements before prompting eliminates 100% of the hallucinations on the RAGTruth conflict examples for GPT-4 and GPT-3.5 Turbo.","keywords":["hallucination elimination","retrieval-augmented generation","noun-phrase collisions","faithfulness","RAGTruth","prompt reformatting","fully-formatted facts","LLM interpretability"],"falsifier":"Apply the same Acurai pipeline to the full RAGTruth corpus or to a fresh sample of RAG queries with irrelevant distractor passages, scoring each response with a blind or automated faithfulness check; if any response contains a statement not supported by the supplied passages, the 100% elimination claim is falsified.","tokens_in":8749,"feed_emoji":"✅","tokens_out":9793,"duration_ms":81955,"temperature":0.7,"pith_summary":"The paper claims that a pre-processing step called Acurai can eliminate 100% of the hallucinations that GPT-4 and GPT-3.5 Turbo produce on the RAGTruth corpus's conflict examples. Acurai rewrites the user's query and the supplied passages before the model sees them: it splits queries that contain semantically similar terms referring to different entities (noun-phrase collisions, such as calcium versus magnesium), converts passages into simple self-contained statements called Fully-Formatted Facts, and remaps colliding references with placeholder tokens. The authors report that this transformation produced faithful answers for all 37 conflict examples across the four GPT-3.5 Turbo and GPT-4 subtle and evident conflict categories, with a 95% Wilson confidence interval of [0.91, 1] for the underlying elimination rate. If the result transfers beyond this small set, it would turn RAG hallucination mitigation into an input-formatting problem rather than a model-capacity problem.","feed_headline":"Reformatting queries and passages eliminates 100% of RAG hallucinations","feed_subtitle":"A pipeline that splits noun-phrase collisions before prompting reports zero hallucinations on RAGTruth conflict sets.","key_machinery":"The paper's central mechanism is the Noun-Phrase Dominance Model, the claim that LLMs organize their internal representations around noun phrases and hallucinate when two semantically similar noun phrases that refer to distinct things are both present in the input. The method works by removing such noun-phrase collisions: it splits the query into separate sub-queries, converts the context into fully-formatted facts (self-contained statements each tied to one subject) so no prompt contains matched pairs of similar terms, and replaces colliding tokens such as citation strings with placeholders that are restored after generation. This machinery is what carries the argument from the calcium/magnesium example to the claimed 100% faithfulness.","core_discovery":"The central discovery is that hallucination in RAG settings can be systematically prevented by restructuring the input so that the model never has to disambiguate two similar noun phrases that denote different things. Acurai operationalizes this with three transformations: splitting the query into sub-queries so that each one contains only non-colliding terms; rewriting the passages into Fully-Formatted Facts, which are simple statements each paired with the query they answer, so that a prompt about magnesium never contains facts about calcium; and replacing references, citations, and other colliding strings with single-token placeholders that are remapped into the output. Applied to the RAGTruth conflict categories, the authors report that the same LLMs that hallucinated on the original inputs produced responses with no deviation from the provided passages in any of the 37 cases.","pith_inferences":["The noun-phrase collision hypothesis suggests that other forms of referential ambiguity — pronouns, numbers, acronyms that overlap with common words — might be handled by the same placeholder-remapping step, but the paper does not test those cases.","Because the evaluation compares outputs manually and without blinding, the 100% figure would be stronger if it were corroborated by an automated faithfulness metric on the same examples.","The method's scope is limited to RAG setups with a handful of relevant passages; testing it with hundreds of passages, as in long-context RAG, would show whether the input-formatting explanation generalizes.","If the method holds outside RAGTruth, it would imply that the 80% accuracy ceiling seen in long-context benchmarks is partly caused by input formatting rather than by model reasoning limits."],"forward_implications":["RAG deployments that supply factual and relevant passages could reach hallucination-free responses by applying the same input transformations, without fine-tuning or model changes.","The query splitter and fact formatting can be run ahead of time: documents can be stored as Fully-Formatted Facts, reducing the latency and cost at request time.","Hallucination prevention replaces post-hoc blocking, so users receive complete answers rather than refusals when the pipeline is used.","The reported 95% confidence interval [0.91, 1] bounds the expected elimination rate under the same conditions for the same model family.","Pairing Acurai with cheaper models such as Llama 3.1 405B could offset the added preprocessing cost, as the paper itself suggests."],"supporting_citations":[{"why":"Supplies the RAGTruth corpus with the four conflict categories used for the evaluation.","marker":"[10]"},{"why":"The long-context RAG benchmark showing that state-of-the-art LLMs do not exceed 80% accuracy, the baseline Acurai aims to beat.","marker":"[6]"},{"why":"Reports sparse interpretable features in LLM internals, used to support the Noun-Phrase Dominance Model.","marker":"[11]"},{"why":"Similar evidence of interpretable features in another LLM, supporting the same architectural premise.","marker":"[12]"},{"why":"The calcium/magnesium example demonstrating that a clear prompt can still produce a cross-subject hallucination.","marker":"[15]"},{"why":"The follow-up example that explicitly ties the calcium/magnesium error to semantically similar noun phrases.","marker":"[16]"},{"why":"Documents the high hallucination rate for PubMed IDs and citation details, motivating the placeholder-remapping step.","marker":"[18]"}],"fun_headline_variants":["Reformatting queries and contexts eliminates 100% of RAG hallucinations","Zero RAG hallucinations by splitting noun-phrase collisions","Acurai input reshaping yields perfect RAG faithfulness","Reformatting before prompting removes all RAG errors","GPT-3.5 and GPT-4 hit zero hallucination after reformatting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim collapses if the 37 conflict examples in the four RAGTruth categories are not representative of RAG hallucinations, or if the authors' manual, unblinded comparison of responses to passages is not a reliable measure of faithfulness.","fun_headline_variants_meta":{"raw":{"variants":["Reformatting queries and contexts eliminates 100% of RAG hallucinations","Zero RAG hallucinations by splitting noun-phrase collisions","Acurai input reshaping yields perfect RAG faithfulness","Reformatting before prompting removes all RAG errors","GPT-3.5 and GPT-4 hit zero hallucination after reformatting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000282,"raw_usage":{"total_tokens":1646,"prompt_tokens":903,"completion_tokens":743,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":657}},"tokens_in":519,"tokens_out":743,"duration_ms":8069,"temperature":1.0,"reasoning_tokens":657,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:50:24.872823+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the same Acurai pipeline to the full RAGTruth corpus or to a fresh sample of RAG queries with irrelevant distractor passages, scoring each response with a blind or automated faithfulness check; if any response contains a statement not supported by the supplied passages, the 100% elimination claim is falsified.","supporting_citations":[{"cited_title":"Long context rag performance of llms, 2024","cited_arxiv_id":null,"evidence_quote":"The long-context RAG benchmark showing that state-of-the-art LLMs do not exceed 80% accuracy, the baseline Acurai aims to beat."},{"cited_title":"Scaling monosemanticity: Extracting inte r- pretable features from claude 3 sonnet, 2024","cited_arxiv_id":null,"evidence_quote":"Similar evidence of interpretable features in another LLM, supporting the same architectural premise."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The calcium/magnesium example demonstrating that a clear prompt can still produce a cross-subject hallucination."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The follow-up example that explicitly ties the calcium/magnesium error to semantically similar noun phrases."},{"cited_title":"Bhattacharyya","cited_arxiv_id":null,"evidence_quote":"Documents the high hallucination rate for PubMed IDs and citation details, motivating the placeholder-remapping step."}],"review_version":1}