{"id":"2a1baeb5-c2e9-49bb-ba6e-a6414af95e2e","arxiv_id":"2412.11728","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SECRET converts long deep-hashing codes into segmented hash codes with hash-table lookup, cutting recall time by over 95% with a small accuracy drop.","lead":"SECRET speeds up deep-hashing code retrieval by cutting hash codes into short segments and using hash-table lookups instead of scanning the whole database. It reports at least 95% less recall time with accuracy within about 2-3% of existing deep hashing models on Python and Java benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table II's ≥95% recall-time reduction is measured against linear-scan baselines running at ~230 MB/s, well below a word-level popcount scan; with a fair baseline the headline advantage likely shrinks from ~36x to a few x at the tested sizes.","rationale":"The reader's weakest assumption (scale extrapolation) is real but secondary: the paper's own LSH row provides an internal calibration, since LSH at 400k collects ~25,000 candidates per query in ~3.9 ms, so even a large candidate-set growth at 45M would plausibly keep SECRET's lookup well below the linear-scan baseline. The primary weakness is the fairness of the baseline itself. Table II's linear-scan baselines move only ~230 MB/s per query; a competent 64-bit popcount scan of L3-resident data would take roughly 0.5-1.5 ms/query at 400k rather than 28 ms, which is comparable to SECRET's own 0.78 ms/query. If so, the 36x speedup and the 'at least 95%' figure shrink to a few x at the measured sizes, even if the method remains genuinely useful at very large databases. I am not claiming fraud or that the method does not scale; I am claiming the empirical magnitude of the headline number is unsupported until the baseline is shown to be near-optimal or the number is re-measured against a near-optimal baseline. Section III-E gives no details about the Hamming-distance loop, and no code is released, so this cannot be checked from the paper. The RQ2 overclaim ('retains more than 98%', Section IV-B, vs. a minimum of 96.7% in Table III, e.g., DJSRH_SECRET with GraphCodeBERT, Python, 128-bit N@10: 0.619/0.640) is real but minor compared to this. Verdict stays CONDITIONAL: acceptance should require re-measurement with an optimized baseline (or an explicit throughput/implementation statement), in addition to the reader's conditions (RQ2 correction, multi-index-hash baseline, code release, variance).","tokens_in":24531,"tokens_out":30499,"duration_ms":286916,"concrete_test":"Re-run the Table II experiment on the same single-threaded Xeon with the deep-hashing linear-scan baselines replaced by a 64-bit XOR+popcount loop (or AVX2 popcount) plus a top-300 min-heap, keeping SECRET's implementation unchanged; check whether the baseline drops by more than ~2-5x (e.g., from 280.1 s toward ~30-100 s at 400k, 128-bit). If it does, recompute all percentage reductions: the 'at least 95%' claim is settled only if every re-measured reduction is still >=95%. As a follow-up within the same run, record the average candidate-set size per query at each N, and if feasible extend to N=2M and 10M to test whether the sublinear trend in Table II persists once bucket occupancy grows.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition for the central claim is the fairness of the linear-scan baseline, and the timings in Table II indicate it is likely unoptimized. In Section IV-A over 10,000 queries, CoSHC/DJSRH/DSAH/JDSH scan the full database per query: at N=50k/100k/200k/400k with 128-bit codes the reported times (31.9/66.4/137.7/280.1 s) correspond to ~250/241/232/229 MB/s of memory read. A 400k-snippet, 128-bit database is only 6.4 MB, well within the L3 cache of the stated Xeon E5-2698v4; a word-level XOR+popcount loop over such data should achieve multiple GB/s, i.e. roughly 0.5-1.5 ms per query before top-300 selection. Nothing in Section III-E states that popcount/SIMD is used, no code is released, and Section VII's threats cover baseline selection but not baseline implementation quality. If the true baseline is 5-15x faster than measured, the headline 'at least 95%' becomes 'roughly 60-90%' at the tested sizes, and the abstract's unqualified 'retrieval time by at least 95%' is wrong. The scale extrapolation the reader identified is genuine but secondary: the LSH row (38.8 s at 400k, ~25,000 candidates/query) shows candidate-heavy lookups cost only a few ms, so a fair baseline is a prerequisite before any extrapolation to the motivating 45M-repo regime can be assessed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SECRET, a framework that accelerates deep-hashing code retrieval by splitting long binary hash codes into short segments, training code and query hashing models to align those segments (via iterative training, adaptive bit relaxing, and dynamic matching objective adjustment), and then recalling candidates through hash-table lookups instead of full linear scans. Experiments on CodeSearchNet Python and Java with CodeBERT/GraphCodeBERT and four deep hashing baselines report that SECRET reduces recall time by at least 95% while largely retaining retrieval accuracy, and that it outperforms LSH under the same number of hash tables. The proposed mechanism is plausible and the ablation study is informative, but the central efficiency claim rests on a baseline implementation whose fairness is not established, and several quantitative statements in the text exceed what the reported tables support.","tokens_in":24991,"tokens_out":5567,"duration_ms":55575,"significance":"If the efficiency claim survives scrutiny, SECRET would be a practically useful and inexpensive acceleration layer for existing deep hashing code retrievers, and the training strategies (iterative alignment, bit relaxing) are interesting beyond the specific application. The paper's strengths are the breadth of the comparison (two languages, two retrieval encoders, four deep hashing baselines, multiple database sizes) and the explicit ablation of the two main design components. The main limitation is that the paper's headline result is an empirical efficiency measurement, and the manuscript currently does not provide enough implementation detail or variance information to verify it; no code is released. The central contribution is therefore promising but not yet established at the level claimed.","major_comments":[{"comment":"The reported linear-scan baselines appear unoptimized in a way that directly affects the headline 'at least 95%' claim. At N=400,000 and 128-bit codes, the database is 6.4 MB; CoSHC's 280.1 s over 10,000 queries corresponds to 28 ms per query, i.e., roughly 229 MB/s of memory read. A single-thread XOR-plus-popcount scan over data that fits in the L3 cache of the stated Xeon E5-2698v4 should achieve multiple GB/s, so the speedup over a fair baseline is likely much smaller than the reported 36x. The paper does not state whether popcount/SIMD is used in the baseline, and no code is released. Please report the exact baseline loop, compiler and flags, and re-measure against a tightly optimized linear scan; this is a prerequisite for the central efficiency claim.","section":"Section III-E and Table II"},{"comment":"The text says SECRET 'retains more than 98%' of performance, but the table's worst relative results are around 96.7%: for example, GraphCodeBERT/DJSRH-SECRET on Python at 128 bits has N@10 = 0.619 versus 0.640 for the baseline, and several other cells are in the 96.7-97.2% range. Please replace the blanket 98% statement with the actual range and rephrase the conclusion so that it matches the data.","section":"Section IV-B and Table III"},{"comment":"All efficiency and effectiveness results are single-run and reported without variance. The effectiveness differences of 0.2-0.4 percentage points that are described as 'outperform' or 'comparable' may be within run-to-run noise. Please report means and standard deviations over multiple seeds (or at least confidence intervals) for the main performance comparisons, and clarify whether the timing numbers are medians over repeated runs.","section":"Tables II and III"},{"comment":"The abstract's unqualified 'at least 95%' reduction and the motivation built on GitHub-scale repositories (45M repositories) are not supported by the tested database sizes (50k-400k), and Section VII explicitly concedes that the data size 'may not be sufficient to demonstrate the performance and efficiency of SECRET under huge databases.' Since lookup cost and candidate-set sorting grow with collision counts, the measured reduction may not extrapolate. Please either add experiments at larger scale or restrict the abstract and conclusions to the tested regime and discuss the scaling behavior of the candidate-set sorting.","section":"Section VII and Abstract"},{"comment":"The paper repeatedly states that SECRET reduces recall time complexity from O(n) to O(1), but Section IV-A later acknowledges that the recall step also counts candidate appearances and sorts the candidate set, with worst-case complexity O(n log n) in the number of snippets. The overview and abstract therefore give an overly strong complexity claim. Please state the actual complexity as lookup plus candidate counting/sorting and describe how it depends on the collision distribution.","section":"Section II-A and Section IV-A"}],"minor_comments":[{"comment":"The cross-references are inconsistent: the text refers to 'Table III-D' and 'Table IV-A' where the displayed tables are Table II and Table III. Please fix all table references.","section":"Section IV-A"},{"comment":"The text refers to 'DASH' when discussing the performance of DSAH; please correct the acronym.","section":"Section IV-B"},{"comment":"The threats section says only 'three deep hashing baselines' were selected, but four are actually evaluated (CoSHC, DJSRH, DSAH, and JDSH).","section":"Section VII"},{"comment":"The symbol k is reused for the segment length in Eq. (2) and for the number of relaxed bits in Eq. (4). Using a distinct symbol (e.g., r) for the relaxation budget would prevent confusion.","section":"Equation (4)"},{"comment":"The sentence 'Since the value range of o_ij is [1, +1]' should presumably read [-1, +1]. Please correct this typo.","section":"Equation (9)"},{"comment":"The storage-cost statement that three relaxed bits make hash segments '2^3 times higher' assumes that each relaxed bit is expanded to both values; please state this assumption explicitly, as the sentence is otherwise easy to misread.","section":"Section V-B"}],"recommendation":"major_revision","confidential_remarks":"The paper leans heavily on CoSHC, the authors' own prior work, as both the primary baseline and the starting point for SECRET. This is not circular in itself, but it raises the bar for showing that the baseline is implemented fairly; the timing anomaly in Table II is exactly where that concern materializes. I would also flag that no code is released, which makes the central efficiency measurement difficult to audit, and that the text contains several factual overstatements that need correction before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is clean: take any deep hashing model for code retrieval, split the long hash codes into shorter segments, build hash tables per segment, and recall candidates by lookup instead of scanning the whole database. The two training tricks (adaptive bit relaxing and dynamic matching objective adjustment) are new and the ablations in RQ3 show they matter. The evaluation is thorough in breadth: four deep hashing baselines, two code encoders, two languages, several database sizes, plus a one-to-many dataset in the appendix. The authors also admit the dataset sizes may not reflect huge real-world repositories, which is honest.\n\nWhere it gets shaky is the efficiency measurement. The reported times for the linear-scan baselines imply a memory throughput of only about 230 MB/s on 128-bit codes. A straightforward XOR-popcount loop should be several times faster on a modern Xeon, and the paper never says what Hamming-distance implementation was used, nor is code released. If the baseline is 5-10x faster, the \"at least 95%\" reduction becomes 60-90% at the tested scales. The sorting cost of the baseline (top-300 selection) may explain part of the gap, but without implementation details the headline number is not yet trustworthy. This is the main thing I'd want fixed before betting on the paper.\n\nSecondary issues: the abstract and conclusion say \"more than 98%\" performance retention, while the worst cell in Table III is 96.7% (and the RQ2 text says \"at least 97.0%\"). That's a real overclaim. There is no multi-index hashing baseline or citation, even though segmented hash tables are exactly multi-index hashing; the novelty is in the deep-hashing training strategy, not the indexing mechanism. No error bars or multiple seeds, and no code or data release, so reproducibility is limited. The scale extrapolation concern is valid and already acknowledged by the authors.\n\nOverall: the paper is a solid engineering contribution to the code-search subfield, not a paradigm shift. The training strategies are interesting, the faithfulness analysis (how many original recalls are preserved) is a nice addition, and the ablation study is genuinely informative. It deserves a serious referee, but the efficiency claim needs a much clearer description of the baseline implementation and ideally a corrected comparison against an optimized scan. I'd send it to peer review with major-revision expectations.","headline":"SECRET is a sensible segmented-hashing wrapper for deep code retrieval, but the headline 'at least 95% faster' leans on a linear-scan baseline whose implementation is too thinly documented to fully trust.","tokens_in":25443,"tokens_out":2902,"would_cite":true,"duration_ms":30503,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SECRET claims that code-candidate recall for deep hashing-based code retrieval can be replaced by hash-table lookups over short hash-code segments, cutting recall time by at least 95% while preserving nearly all retrieval accuracy.","keywords":["code retrieval","deep hashing","segmented hashing","hash table lookup","code search acceleration","iterative training","adaptive bit relaxing"],"falsifier":"Measure SECRET's wall-clock recall time as the code database grows from 400,000 to several million snippets at fixed segment length and candidate budget; if time scales linearly with database size or with the number of hash collisions per segment, the claimed lookup advantage is not realized at the scale the paper targets.","tokens_in":24383,"feed_emoji":"⚡","tokens_out":4687,"duration_ms":41442,"temperature":0.7,"pith_summary":"SECRET is an acceleration layer for deep hashing-based code retrieval. It takes the long binary hash codes produced by existing deep hashing models, splits them into short segments, and builds one hash table per segment; at query time it recalls candidates by table lookups instead of scanning the entire code database and computing Hamming distances. The paper claims this reduces recall time by at least 95% at database sizes from 50,000 to 400,000 snippets while retaining at least 97% of retrieval performance, and that it outperforms the classical hash-table baseline LSH under the same number of hash tables. To make the hash tables effective, SECRET trains the code and query hashing models iteratively so their segment outputs align, relaxes hard-to-align bits to an unknown state, and adjusts training objectives to avoid hash collisions with negative samples.","feed_headline":"Segmented hash codes cut code search recall time by 95 percent","feed_subtitle":"Splitting deep hashes into table lookups replaces linear scanning, keeping retrieval accuracy almost intact.","key_machinery":"The central object is the segmented hash code: a long binary hash code split into short chunks, each of which becomes a key in its own lookup table. A query recalls a code snippet if any of the query's segment keys collides with the snippet's corresponding segment key, so recall cost shifts from an $O(n)$ Hamming scan to $O(1)$-style table lookups followed by counting and sorting the hit candidates. The training machinery that makes collisions meaningful is the iterative alignment loop: freeze one modality's hashing model, use its segment outputs as targets for the other, then alternate; adaptive bit relaxing replaces low-magnitude output bits with an unknown state so hard-to-align bits do not force mismatches; and dynamic matching objective adjustment edits the target labels to steer away from hash values that collide with negative samples in the batch.","core_discovery":"The central claim is that candidate recall for deep-hashing code retrieval can be transformed from a linear Hamming-distance scan of the whole code database into a set of hash-table lookups over segmented hash codes, with no material loss in accuracy. SECRET splits each long hash code into short segments (16 bits in the experiments), builds a lookup table per segment, and retrieves any code snippet whose segment collides with the query's corresponding segment. The iterative training stage alternately freezes one modality's hashing model and uses its output as the training objective for the other, while adaptive bit relaxing marks low-confidence bits as unknown and dynamic matching objective adjustment shifts alignment targets away from negative samples' hash values. On CodeSearchNet-derived Python and Java benchmarks with 128- and 256-bit codes, SECRET reports at least 95% recall-time reduction across all tested database sizes and hash lengths, retains roughly 97-98% of the original deep hashing performance, and matches or beats the base methods on several 256-bit settings.","pith_inferences":["If the 95% figure degrades when hash tables fill with collisions, the method's practical ceiling may be reached well below GitHub-scale corpora; a direct measurement of candidate-list growth versus database size would make this precise.","The adaptive bit relaxing strategy effectively turns binary codes into ternary codes inside the table, so the framework could connect to ternary hashing literature and to learned approximate nearest neighbor indexes that tolerate wildcard bits.","A testable extension would be to apply SECRET to retrieval-augmented code generation or very large monorepo search, where recall latency is often the bottleneck and the re-ranking model is itself expensive.","The paper does not report end-to-end latency including the encoding step; the 95% figure covers only the recall step, so a full-system study could change the practical speedup picture."],"forward_implications":["If the 95%+ recall-time reduction holds at scale, deep hashing code retrieval can serve interactive code search on very large repositories without re-ranking against the entire database.","Because longer hash codes improve SECRET's recall (more segments, more tables), performance lost through segmentation can be recovered by increasing code length at modest storage cost.","SECRET's candidate set is smaller than a full-scan recall set, so the re-rank stage does less work, implying end-to-end latency gains beyond the recall step.","The framework is portable: it wraps the output of any deep hashing model, so future improvements in deep hashing encoders should inherit the same speedup without redesign.","The sublinear growth of SECRET's recall time as database size grows suggests the efficiency advantage widens as the code database expands."],"supporting_citations":[{"why":"Supplies CoSHC, the deep hashing-based recall pipeline with linear Hamming-distance scanning that SECRET accelerates.","marker":"[13]"},{"why":"Supplies CodeBERT as the base code-query retrieval model and the Python and Java dataset pairs used in evaluation.","marker":"[18]"},{"why":"Supplies GraphCodeBERT as the second base retrieval model whose deep hashing outputs SECRET segments.","marker":"[19]"},{"why":"Provides CodeSearchNet, the source of the code-query training and test data.","marker":"[10]"},{"why":"Provides LSH, the classical hash-table baseline compared against SECRET under the same number of hash tables.","marker":"[23]"},{"why":"Provides DJSRH, a deep cross-modal hashing baseline whose output SECRET wraps.","marker":"[20]"},{"why":"Provides DSAH, a deep hashing baseline whose output SECRET wraps.","marker":"[21]"},{"why":"Provides JDSH, a deep hashing baseline whose output SECRET wraps.","marker":"[22]"}],"fun_headline_variants":["Segmented deep hashing cuts code recall time by 95%","Hash-table lookups replace linear scan for faster code retrieval","SECRET: split hash codes for 20x faster code recall","Segment hashes reduce code search recall time by at least 95%","Iterative hash segmentation accelerates code retrieval with little accuracy loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim assumes that the sublinear recall-time speedups measured on databases of up to 400,000 snippets will continue to hold on the tens-of-millions-scale code databases that motivate the work, an extrapolation the paper itself flags as untested.","fun_headline_variants_meta":{"raw":{"variants":["Segmented deep hashing cuts code recall time by 95%","Hash-table lookups replace linear scan for faster code retrieval","SECRET: split hash codes for 20x faster code recall","Segment hashes reduce code search recall time by at least 95%","Iterative hash segmentation accelerates code retrieval with little accuracy loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000729,"raw_usage":{"total_tokens":3297,"prompt_tokens":1007,"completion_tokens":2290,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":2202}},"tokens_in":623,"tokens_out":2290,"duration_ms":17167,"temperature":1.0,"reasoning_tokens":2202,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:38:22.635693+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure SECRET's wall-clock recall time as the code database grows from 400,000 to several million snippets at fixed segment length and candidate budget; if time scales linearly with database size or with the number of hash collisions per segment, the claimed lookup advantage is not realized at the scale the paper targets.","supporting_citations":[{"cited_title":"Accelerating code search with deep hashing and code classification,","cited_arxiv_id":null,"evidence_quote":"Supplies CoSHC, the deep hashing-based recall pipeline with linear Hamming-distance scanning that SECRET accelerates."},{"cited_title":"Graphcodebert: Pre-training code representations with data flow,","cited_arxiv_id":null,"evidence_quote":"Supplies GraphCodeBERT as the second base retrieval model whose deep hashing outputs SECRET segments."}],"review_version":1}