The reviewed record of science sign in
Pith

arxiv: 2607.01601 · v1 · pith:F6YNIRJ7 · submitted 2026-07-02 · cs.AI

SemHash-LLM: A Multi-Granularity Semantic Hashing Framework for Document Deduplication

Xinyi Fang , Kejian Tong , Jiabei Liu , Tao Ning , Yuhang He This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-03 14:52 UTCgrok-4.3pith:F6YNIRJ7record.jsonopen to challenge →

classification cs.AI
keywords document deduplicationsemantic hashingmulti-granularityMinHashLLM efficiencycascaded filteringcontrastive boundary learning
0
0 comments X

The pith

SemHash-LLM detects document duplicates at high quality using under one percent neural verification cost.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces SemHash-LLM as a multi-granularity framework for large-scale document deduplication that preserves semantic equivalence while keeping efficiency high over massive corpora. It unifies semantic projection hashing in distilled LLM embeddings, attention-weighted MinHash, contrastive boundary learning, and selective LLM adjudication. Signals from character, token, and document levels are combined through gated fusion before a cascaded filtering pipeline reduces candidates. Adaptive decision boundaries and uncertainty estimation are added to handle template pollution, short text changes, containment, and viral fragments. The central experimental result is that this pipeline delivers strong detection quality while limiting neural verification to less than one percent of cases.

Core claim

SemHash-LLM learns compact binary codes in distilled LLM embedding space, applies attention-weighted MinHash to suppress boilerplate, fuses multi-level signals via gating, and routes uncertain cases through cascaded filters with adaptive boundaries, thereby achieving strong duplicate detection quality across varied text perturbations with less than one percent neural verification cost.

What carries the argument

Gated fusion of character-, token-, and document-level signals combined with cascaded filtering, attention-weighted MinHash, and adaptive decision boundaries for candidate reduction before selective LLM adjudication.

Load-bearing premise

The assumption that gated fusion of character, token, and document signals plus cascaded filtering and adaptive boundaries will reliably preserve semantic equivalence detection across template pollution, short text perturbation, containment, and viral fragments without unacceptable false positives or negatives.

What would settle it

A controlled experiment on a diverse corpus where full-LLM verification is run on all pairs and the method's precision or recall falls substantially below the full-LLM baseline while still claiming under one percent neural calls.

Figures

Figures reproduced from arXiv: 2607.01601 by Jiabei Liu, Kejian Tong, Tao Ning, Xinyi Fang, Yuhang He.

Figure 1
Figure 1. Figure 1: Overview of the SemHash-LLM framework. Documents are encoded [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Semantic Projection Hashing. A student encoder distilled from the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Attention-Weighted MinHash pipeline. Aggregated multi-head at [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Contrastive Boundary Learning and LLM-as-Judge refinement. The [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Cascaded filtering pipeline. Four stages—Bloom filter, semantic hash [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
read the original abstract

Large scale document deduplication must preserve semantic equivalence while remaining efficient over massive corpora. We present SemHash LLM, a multi granularity framework that unifies semantic projection hashing, attention weighted MinHash, contrastive boundary learning, and selective LLM based adjudication. The method combines character, token, and document level signals through gated fusion, then applies a cascaded filtering pipeline for efficient candidate reduction. Semantic projection hashing learns compact binary codes in distilled LLM embedding space, while attention weighted Min- Hash suppresses boilerplate and emphasizes informative content. Adaptive decision boundaries and uncertainty estimation further improve robustness across template pollution, short text perturbation, containment, and viral fragments. Experiments show that SemHash LLM achieves strong duplicate detection quality with less than one percent neural verification cost.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The paper introduces SemHash-LLM, a multi-granularity semantic hashing framework for large-scale document deduplication. It unifies semantic projection hashing in distilled LLM embeddings, attention-weighted MinHash, contrastive boundary learning, gated fusion of character/token/document signals, cascaded filtering, adaptive boundaries, and selective LLM adjudication to preserve semantic equivalence while reducing neural verification cost to under 1%. Experiments are claimed to demonstrate strong duplicate detection quality across challenges like template pollution and viral fragments.

Significance. If the experimental claims hold with proper validation, the framework could offer a practical advance in efficient deduplication for massive corpora by combining hashing efficiency with semantic robustness, potentially reducing computational overhead in LLM-based pipelines. The multi-level signal fusion and cascaded approach address real scalability needs, though this remains conditional on the missing empirical support.

major comments (1)
  1. Abstract: The central claim that 'Experiments show that SemHash LLM achieves strong duplicate detection quality with less than one percent neural verification cost' is presented without any supporting data, baselines, metrics, dataset descriptions, ablation studies, or error bars. This absence makes it impossible to assess whether the gated fusion, attention-weighted MinHash, or adaptive boundaries deliver the claimed performance, rendering the experimental contribution unverifiable from the provided text.
minor comments (1)
  1. Abstract: Terminology such as 'contrastive boundary learning' and 'semantic projection hashing' is introduced without definitions or references to prior work, which could be clarified for readers unfamiliar with the subfield.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their review. The major comment concerns the abstract's claim lacking supporting details. We clarify that the full manuscript (as referenced in the source context) contains the complete experimental sections with datasets, baselines, metrics, ablations, and results.

read point-by-point responses

  1. Referee: [—] Abstract: The central claim that 'Experiments show that SemHash LLM achieves strong duplicate detection quality with less than one percent neural verification cost' is presented without any supporting data, baselines, metrics, dataset descriptions, ablation studies, or error bars. This absence makes it impossible to assess whether the gated fusion, attention-weighted MinHash, or adaptive boundaries deliver the claimed performance, rendering the experimental contribution unverifiable from the provided text.

    Authors: Abstracts are concise summaries by design and do not include full data, baselines, or ablations; those appear in the dedicated Experiments section of the full manuscript. The manuscript details the evaluation across template pollution, viral fragments, and other challenges, with comparisons to baselines, metrics such as precision/recall/F1, ablation studies on gated fusion/attention-weighted MinHash/adaptive boundaries, and reported neural verification costs under 1% with error bars. The experimental contribution is therefore verifiable from the complete manuscript. revision: no

Circularity Check

0 steps flagged

No significant circularity

full rationale

The provided abstract and description contain no equations, parameter fits, derivations, or self-citations that could reduce any claimed result to its own inputs by construction. The framework is presented as a combination of existing techniques (semantic projection hashing, attention-weighted MinHash, gated fusion, cascaded filtering) with experimental outcomes reported separately. No load-bearing step matches any of the enumerated circularity patterns; the central claims rest on empirical evaluation rather than internal redefinition or self-referential prediction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides no information on free parameters, axioms, or invented entities; all arrays are therefore empty.

pith-pipeline@v0.9.1-grok · 5665 in / 1062 out tokens · 34044 ms · 2026-07-03T14:52:49.308150+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages · 2 internal anchors

  1. [1]

    Deduplicating training data makes language models better,

    K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini, “Deduplicating training data makes language models better,” inProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 8424– 8445

    work page 2022

  2. [2]

    Resilient Routing: Risk-Aware Dynamic Routing in Smart Logistics via Spatiotemporal Graph Learning

    Z. Xue, S. Zhao, Y . Qi, X. Zeng, and Z. Yu, “Resilient routing: Risk- aware dynamic routing in smart logistics via spatiotemporal graph learning,”arXiv preprint arXiv:2601.13632, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  3. [3]

    The Pile: An 800GB Dataset of Diverse Text for Language Modeling

    L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshimaet al., “The pile: An 800gb dataset of diverse text for language modeling,”arXiv preprint arXiv:2101.00027, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  4. [4]

    A unified inference optimization framework for qwen3-vl-2b- instruct

    Q. Zhou, “A unified inference optimization framework for qwen3-vl-2b- instruct.”

    work page

  5. [5]

    Dolma: An open corpus of three trillion tokens for language model pretraining research,

    L. Soldaini, R. Kinney, A. Bhagia, D. Schwenk, D. Atkinson, R. Authur, B. Bogin, K. Chandu, J. Dumas, Y . Elazaret al., “Dolma: An open corpus of three trillion tokens for language model pretraining research,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 15 725– 15 788

    work page 2024

  6. [6]

    Coil: Revisit exact lexical match in information retrieval with contextualized inverted list,

    L. Gao, Z. Dai, and J. Callan, “Coil: Revisit exact lexical match in information retrieval with contextualized inverted list,” inProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 3030–3042

    work page 2021

  7. [7]

    Colbertv2: Effective and efficient retrieval via lightweight late interac- tion,

    K. Santhanam, O. Khattab, J. Saad-Falcon, C. Potts, and M. Zaharia, “Colbertv2: Effective and efficient retrieval via lightweight late interac- tion,” inProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022, pp. 3715–3734

    work page 2022

  8. [8]

    Simcse: Simple contrastive learning of sentence embeddings,

    T. Gao, X. Yao, and D. Chen, “Simcse: Simple contrastive learning of sentence embeddings,” inProceedings of the 2021 conference on empirical methods in natural language processing, 2021, pp. 6894–6910

    work page 2021

  9. [9]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xinget al., “Judging llm-as-a-judge with mt-bench and chatbot arena,”Advances in neural information processing systems, vol. 36, pp. 46 595–46 623, 2023

    work page 2023

  10. [10]

    G-eval: Nlg evaluation using gpt-4 with better human alignment,

    Y . Liu, D. Iter, Y . Xu, S. Wang, R. Xu, and C. Zhu, “G-eval: Nlg evaluation using gpt-4 with better human alignment,” inProceedings of the 2023 conference on empirical methods in natural language processing, 2023, pp. 2511–2522

    work page 2023

  11. [11]

    Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models,

    P. Manakul, A. Liusie, and M. Gales, “Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models,” inProceedings of the 2023 conference on empirical methods in natural language processing, 2023, pp. 9004–9017

    work page 2023

  12. [12]

    Prism: Pipeline for root-cause investigation via specialized multi-agents,

    W. Yan, Y . Wu, P. Liang, M. Yuan, J. Liu, J. Yang, and X. Li, “Prism: Pipeline for root-cause investigation via specialized multi-agents,” in 2026 International Conference on Generative Artificial Intelligence and Information Security (GAIIS). IEEE, 2026, pp. 709–712

    work page 2026

  13. [13]

    Dynarag: Bridging static and dynamic knowledge in retrieval-augmented genera- tion,

    P. Liang, M. Yuan, J. Liu, J. Yang, X. Li, W. Yan, and Y . Wu, “Dynarag: Bridging static and dynamic knowledge in retrieval-augmented genera- tion,” in2026 9th International Symposium on Big Data and Applied Statistics (ISBDAS). IEEE, 2026, pp. 442–445

    work page 2026

  14. [14]

    Pyramid convolution and bidirectional graph attention for cyber threat detection from unstructured text,

    Y . Xu, “Pyramid convolution and bidirectional graph attention for cyber threat detection from unstructured text,” inProceedings of the 2026 International Conference on Artificial Intelligence and Control, 2026, pp. 561–567

    work page 2026

  15. [15]

    Roofline-guided mixed quantization and kernel co- optimization for efficient large language model inference on arm cpus

    Q. Zhou, “Roofline-guided mixed quantization and kernel co- optimization for efficient large language model inference on arm cpus.”

    work page