{"id":"e56d7742-f738-4db3-a8a8-0938dc736fdd","arxiv_id":"2504.17519","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Generative retrieval with text-based docids (n-grams, titles, URLs) generalizes to newly added documents far better than numeric docids, and a constrained multi-docid numeric design recovers much of the gap on NQ.","lead":"Generative retrieval models that write document identifiers instead of matching against an index have been tested mostly on static corpora; this paper tests them when new documents keep arriving. It finds that text-based identifiers generalize much better than numeric ones without retraining, then proposes a faster hybrid numeric design that is competitive on one benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The dynamic-corpus simulation is a random i.i.d. split; under realistic distribution drift the claimed text-docid generalization advantage may not transfer, a limitation the paper's own IDBI remark acknowledges.","rationale":"The strongest part of the paper is the consistent large gap between text-based and numeric-based docids in GAn on both NQ and MS-MARCO, and the efficiency analysis for MDGR. Those internal comparisons are likely robust. The load-bearing weak point is the benchmark's external validity: Section 3.1 randomizes which documents are 'new', which removes exactly the covariate shift that makes real dynamic corpora hard. If the text-docid advantage is driven by lexical overlap, as the paper's n-gram analysis suggests, then a temporally or topically drifted split is the critical stress test. The paper's own Remark after Definition 3 acknowledges the same-distribution assumption for IDBI, so this is not an invented objection. The concern does not overturn the text-vs-numeric finding on the paper's own benchmark, but it does mean the abstract's unqualified 'dynamic corpora' claim and the MDGR generalization claim are conditional until tested under drift. A secondary supporting observation: on the paper's own MS-MARCO split, the best text-based models have GAn 0.586-0.600, below BM25's 0.618, so the 'surpassing BM25' part of the headline is already dataset-dependent. The reader's conditional verdict remains appropriate; no change is needed.","tokens_in":20354,"tokens_out":7302,"duration_ms":72083,"concrete_test":"Construct a non-i.i.d. dynamic split of NQ by ordering documents by Wikipedia creation/first-revision date (or, for MS-MARCO, by URL-domain or topic clusters) so D0 is the earliest 50% and D1-D5 are successive slices; keep the paper's official implementations and hyperparameters, train only on D0, then recompute GAn for each method. If text-based models (SEAL, LTRGR) remain clearly above numeric models (DSI-SE, Ultron-PQ) and near DPR-HN, the concern does not land. If the gap shrinks or reverses—note that on the existing MS-MARCO random split LTRGR (0.600) and SEAL (0.586) already fall below BM25 (0.618)—then the 'surpassing BM25' and 'comparable to dense retrieval' claims must be qualified to same-distribution additions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 builds the dynamic benchmark by randomly selecting 50% of a static dataset as D0 and partitioning the remaining 50% into five equal chunks (D1-D5). This makes the newly added documents independent random draws from the same distribution as the training corpus. The paper's central claim—that frozen text-docid GR models generalize to unseen documents and that MDGR narrows the numeric/text gap—requires this simulation to represent real dynamic corpora, but real corpora exhibit temporal order, topic drift, style shifts, and new vocabulary. The mechanism the paper invokes for text-based success, semantic familiarity and n-gram lexical overlap, is precisely the property that random splits maximize: new documents share the same vocabulary and topic distribution, so a frozen model can retrieve them by regenerating familiar substrings. Under drift, that overlap shrinks, and the advantage could weaken or reverse. The paper itself flags the same distributional assumption in the Remark after Definition 3 of IDBI: 'The IDBI assumes that initial documents D0 and new documents Dnew are drawn from the same latent query-document relevance distribution.' Because the main empirical comparison and the IDBI-based bias analysis both rely on this assumption, it is load-bearing rather than cosmetic.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a reproducibility and analysis study of generative retrieval (GR) models over dynamic corpora. It partitions NQ and MS-MARCO into an initial set D0 and five incremental chunks D1–D5, trains frozen GR models on D0 only, and compares text-based docid models (SEAL, MINDER, LTRGR, Ultron-URL) against numeric-based docid models (DSI-SE, Ultron-PQ, NCI, GenRET), with BM25, DPR, and DPR-HN as sparse/dense baselines. The main empirical findings are that text-based docids generalize better to newly added documents, that finer-grained and more lexically diverse docids help, and that numeric-based docids suffer from a bias toward the initial document set. The paper then proposes MDGR, a numeric-based multi-docid method with chunk-level product quantization, constrained docid expansion, and a coverage-plus-rank inference score, and reports results on NQ showing that it narrows the gap between numeric efficiency and text effectiveness. A secondary experiment compares DSI and SEAL with and without DSI++-style incremental training.","tokens_in":20543,"tokens_out":5026,"duration_ms":51955,"significance":"The study is useful and timely: it independently reproduces several GR systems, provides a systematic comparison under a common dynamic-corpus protocol, and offers a plausible explanation of numeric-docid failure via the IDBI bias measure. The paper ships code, follows official implementations for baselines, and makes falsifiable claims about docid granularity and lexical diversity. If the results hold, the finding that frozen text-based GR models can retrieve newly added documents without retraining, and that MDGR preserves much of this capability with numeric efficiency, is a meaningful contribution to the GR literature. The main risks are that the dynamic setting is simulated by an i.i.d. random split, MDGR is evaluated on only one dataset, and the statistical significance statement is not backed by experimental details.","major_comments":[{"comment":"The statistical validation section states that all results in Tables 1 and 2 achieved significance at p < 0.05, but it does not report the test used, the number of random seeds or runs, or the standard deviation of the reported Hit@10 values. Because several comparisons in Tables 1, 2, and 4 involve small margins (for example, LTRGR vs. MINDER on NQ initial documents), this assertion is not verifiable. Please provide the test procedure, variance across runs, and run counts.","section":"§3.4"},{"comment":"The dynamic-corpus benchmark is constructed by randomly splitting a static dataset into a 50% initial set and five 10% chunks, which makes the newly added documents i.i.d. draws from the same distribution as D0. This is the regime most favorable to the paper's 'semantic familiarity' mechanism, since new documents share the vocabulary and topic distribution of the training set. The paper's own remark after Definition 3 acknowledges that IDBI assumes initial and new documents come from the same latent relevance distribution, and the same assumption underlies the main generalization comparison. The central claim that text-based docids generalize better to unseen documents is therefore only validated under no distribution drift. Please add a shifted or temporally ordered split (e.g., ordering by date or by vocabulary novelty) or explicitly restrict the conclusions to i.i.d. corpus growth.","section":"§3.1 and Definition 3 Remark"},{"comment":"MDGR, the paper's proposed method, is evaluated only on the NQ dataset; no MS-MARCO results are reported for it. Since the paper's main empirical claims are established on two datasets and MDGR is presented as a general framework, the single-dataset evaluation is insufficient to support the claimed 'improved performance in dynamic corpus.' Please report MDGR on MS-MARCO or substantially temper the contribution claim.","section":"§6.3, Table 6"},{"comment":"The docid vocabulary size k=1024 for MDGR is selected from a sweep over {64, 256, 1024, 4096, 8192} performed on the same NQ test sets that are later used for the final comparison in Table 6. It is unclear whether the selection was made on a held-out portion or with knowledge of the test results, so the final MDGR numbers may include an optimistic model-selection effect. Please clarify the model-selection protocol and report sensitivity of the main conclusion to k.","section":"§6.1, Figure 3"}],"minor_comments":[{"comment":"The semantic familiarity S is defined but never directly estimated; the paper infers 'alignment' indirectly from docid type and IDBI. Please state whether S is operationalized or used only as a conceptual device.","section":"§5.1, Eq. (6)"},{"comment":"There is a typo in 'ULtron variants' in the implementation details; it should read 'Ultron variants'.","section":"§3.2"},{"comment":"The code availability line says 'available at here' without a visible URL; please provide a working link or repository identifier.","section":"Abstract/footnote"},{"comment":"In the manuscript PDF, the figures contain garbled 'uni000...' token sequences in axis labels and legends, making them hard to interpret. Please ensure the figures are rendered with readable text.","section":"Figures 1–3"},{"comment":"The inference score depends on the hyperparameter beta, but the paper never reports the value used in Table 6 or a sensitivity analysis for beta.","section":"§6.2, Eq. (8)"},{"comment":"The table reports 'Memory', 'Tok-K', and 'Latency' without defining the measurement setup; please specify whether memory is model size only or includes the index, and describe the hardware and batch size used for latency.","section":"Table 7"},{"comment":"The numeric docid size range '32–10,000' is ambiguous; please clarify whether this refers to cluster count per subvector, total docid vocabulary size, or something else.","section":"Table 5"},{"comment":"The comparison between DSI and DSI++ and between SEAL and SEAL++ is interesting but lacks significance information; the same statistical details requested for Tables 1 and 2 should be supplied here as well.","section":"§4, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is the camera-ready version of a SIGIR 2025 paper; my report focuses on internal validity. The most fixable load-bearing issues are the missing MS-MARCO evaluation for MDGR and the vague significance statement. The i.i.d. split limitation is a genuine scope restriction, but it would be acceptable if the paper explicitly framed its conclusions as applying to corpus growth without distribution drift rather than to general real-world dynamic corpora."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time if you work on generative retrieval. The paper does a simple thing the subfield needed: it takes eight existing GR models, freezes their parameters, adds documents in five chunks, and measures retrieval on old and new documents under a common protocol. The main finding is large and consistent across NQ and MS-MARCO: text-based docids (SEAL, LTRGR, MINDER) retain most of their ability to retrieve newly added documents, while numeric-based docids (DSI-SE, Ultron-PQ, NCI, GenRET) drop sharply. Gaps that size are not implementation quirks. The paper also earns credit for isolating three plausible causes—semantic familiarity, docid granularity, lexical diversity—and backing the granularity story with an ablation. The proposed MDGR is a legitimate attempt to get numeric-based efficiency with text-like generalization: chunk-level PQ docids, constrained expansion onto the existing docid vocabulary, and coverage-plus-rank scoring. On NQ it sits between SEAL and DPR-HN and is much faster than SEAL. That is a real contribution.\n\nSoft spots, in descending order. First, the dynamic simulation is an i.i.d. random split: 50% initial, five random 10% chunks. That assumes new documents come from the same distribution as the training corpus. Real corpora drift, and the paper's own Remark after Definition 3 admits the IDBI assumes the same latent query-document relevance distribution. The text-docid advantage is tied to lexical overlap with pretraining; under topic drift or new vocabulary, that advantage could weaken. This does not sink the main comparison, but the title and abstract overstate by saying 'dynamic corpora' when the evidence is for random-chunk dynamics. Second, Section 3.4 claims p<0.05 for all results in Tables 1 and 2 but gives no test, no variance, no seeds—a concrete reproducibility gap. Third, MDGR is evaluated only on NQ, while the discussion frames it as general, and k=1024 was selected from a sweep on the same NQ evaluation in Table 6—mild selection on the test set. The code link is a bare 'here' with no URL, annoying in a replication paper.\n\nThe citation pattern is honest; prior dynamic-corpus work is cited, and the GenRET baseline uses the official implementation, so no circularity. The central empirical claim holds up on this benchmark. I would not desk reject. The paper deserves a serious referee; the main asks are a drift-aware evaluation, real statistical detail, and an MS-MARCO run for MDGR.","headline":"Useful and honest replication study: text-based docids clearly generalize better than numeric ones on random dynamic splits, but the i.i.d. benchmark and single-dataset MDGR evaluation mean the sweeping 'dynamic corpora' claims need tempering.","tokens_in":21168,"tokens_out":5396,"would_cite":true,"duration_ms":47064,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The format of a document's identifier decides whether a generative retriever can find documents added after training.","keywords":["generative retrieval","dynamic corpora","docid design","text-based docids","numeric-based docids","retrieval generalization","multi-docid retrieval","information retrieval"],"falsifier":"Run the same protocol with chronological or topic-based splits, where the five added chunks come from later periods with new vocabulary; if text-docid models no longer beat BM25 and dense retrieval on those chunks, the central claim does not extend beyond the random-split assumption.","tokens_in":20104,"feed_emoji":"🔎","tokens_out":7202,"duration_ms":67452,"temperature":0.7,"pith_summary":"This paper tries to establish that, in generative retrieval, the way a document's identifier (docid) is written determines whether a trained model can retrieve documents added after training. Re-running eight generative retrieval models on dynamic splits of NQ and MS MARCO, it finds that models using text-based docids—titles, URLs, n-grams, or multi-text identifiers—generalize to newly added documents without retraining, while numeric-docid models perform much worse on new documents and keep favoring the original collection. The paper argues that text docids win because they align with the language model's pretraining distribution, encode documents at a finer granularity, and use a larger effective vocabulary. It then proposes MDGR, a multi-docid numeric scheme that reuses existing numeric docids for new document chunks and combines retrieval speed with text-level generalization. If correct, the result gives search-engine builders a cheap path to updating generative retrieval over evolving collections.","feed_headline":"Text-like docids keep generative retrieval working on fresh documents","feed_subtitle":"N-gram and multi-text docids beat BM25 on fresh collections; numeric docids collapse without retraining.","key_machinery":"The load-bearing object is the docid, the sequence of tokens a generative retriever is trained to emit for each document. The paper separates docids into numeric-based types, such as cluster or quantized index codes used by DSI-SE, Ultron-PQ, and NCI, and text-based types, such as titles, URLs, n-grams, and multi-text identifiers used by SEAL, MINDER, and LTRGR; the argument hinges on how each type behaves when the prefix tree is extended with docids for new documents. The supporting analytical tool is the Initial Document Bias Index (IDBI), which measures whether Top-K results contain more initial documents than their corpus share would predict. The proposed method is MDGR, a multi-docid design that chunks documents, maps each chunk to an existing numeric docid via product quantization, and ranks documents by coverage and beam-search rank of the generated docids.","core_discovery":"On the paper's own terms, the central discovery is that generalization to unseen documents in generative retrieval is governed by the docid representation rather than by model size or training objective. Using a protocol that trains each model on half of a corpus and then incrementally indexes the remaining half in five chunks, the paper reports that text-based docid models such as SEAL and LTRGR keep Hit@10 on new documents at 0.725–0.799 on NQ, comparable to or better than DPR-HN (0.632) and BM25 (0.586), while numeric-docid models fall to 0.209–0.542. The paper attributes the gap to three properties of text docids: semantic familiarity with the pretrained language model, fine-grained multi-docid coverage, and high lexical diversity. It also introduces the Initial Document Bias Index (IDBI), showing that numeric docids over-retrieve initial documents, and it proposes MDGR, which assigns frozen numeric docids to chunks of new documents under constrained expansion, preserving efficiency while improving dynamic-corpus performance over plain numeric baselines.","pith_inferences":["If the semantic-familiarity explanation is right, the advantage of text docids should scale with the quality of the underlying language model's pretraining; a model pretrained on unrelated text should show less benefit from n-gram docids.","The IDBI could be repurposed as a diagnostic for continual learning and forgetting in sequence-to-sequence models beyond retrieval, since it isolates bias toward earlier training items.","A direct extension would use temporal or topic-drift splits: the paper's random-split protocol assumes new documents come from the same distribution, so ordered by time the text-docid advantage may shrink or reverse.","MDGR's constrained reuse of old docids suggests a broader principle: in generative retrieval, novelty is handled by composing familiar tokens rather than minting new codes, so token-vocabulary closure may matter more than docid semanticity."],"forward_implications":["Frozen generative retrieval models can serve a growing corpus at index-update cost only, if their docids are text-like and fine-grained; no retraining is needed for new documents.","Numeric-docid GR models, as currently designed, cannot be trusted for dynamic collections without continual learning; their generation bias toward the training corpus is measurable with IDBI.","Docid design becomes a first-class decision in GR deployment: finer granularity and larger effective vocabulary translate directly into generalization on unseen documents.","MDGR-style constrained expansion shows that a numeric docid model can narrow most of the generalization gap to text docids while keeping the storage and decoding speed of numeric codes.","Existing continual-learning gains for numeric-docid models come with forgetting on initial documents, whereas text-docid models already have most of the benefit without retraining."],"supporting_citations":[{"why":"DSI-SE supplies the leading numeric-docid baseline whose retrieval of newly added documents collapses, motivating the central generalization claim.","marker":"[30]"},{"why":"Ultron-PQ and Ultron-URL provide the numeric and URL-docid contrasts showing efficiency but poor dynamic-corpus adaptation.","marker":"[36]"},{"why":"NCI is another numeric-docid generative retrieval model used to confirm the systematic failure pattern on unseen documents.","marker":"[31]"},{"why":"SEAL is the primary text-based n-gram docid model used to demonstrate strong generalization without retraining.","marker":"[2]"},{"why":"MINDER provides a multi-text docid baseline that outperforms plain numeric docids on dynamic corpora.","marker":"[15]"},{"why":"LTRGR is the best-performing text-based model in the comparison, achieving the highest generalization scores on new documents.","marker":"[16]"},{"why":"BM25 is the sparse retrieval baseline that text-docid generative models surpass on newly added NQ documents.","marker":"[26]"},{"why":"DPR is the dense retrieval baseline whose frozen-encoder performance on new documents is matched by the best text-docid generative models.","marker":"[10]"},{"why":"NQ is one of the two benchmark datasets used to construct the dynamic-corpus simulation.","marker":"[13]"},{"why":"MS MARCO is the other benchmark dataset used to construct the dynamic-corpus simulation.","marker":"[1]"}],"fun_headline_variants":["Docid format decides generative retrieval on fresh docs","Text docids keep generative retrieval sharp on new data","Numeric docids fail on fresh corpora; text docids thrive","Generative retrieval generalization hinges on docid type","Docid representation is the key to fresh-corpus generative retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The simulation treats newly added documents as random chunks of the same static collection, so the tests do not include real-world drift in topic, style, or vocabulary over time.","fun_headline_variants_meta":{"raw":{"variants":["Docid format decides generative retrieval on fresh docs","Text docids keep generative retrieval sharp on new data","Numeric docids fail on fresh corpora; text docids thrive","Generative retrieval generalization hinges on docid type","Docid representation is the key to fresh-corpus generative retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000553,"raw_usage":{"total_tokens":2699,"prompt_tokens":1070,"completion_tokens":1629,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":686,"completion_tokens_details":{"reasoning_tokens":1549}},"tokens_in":686,"tokens_out":1629,"duration_ms":10502,"temperature":1.0,"reasoning_tokens":1549,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:38:09.561676+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same protocol with chronological or topic-based splits, where the five added chunks come from later periods with new vocabulary; if text-docid models no longer beat BM25 and dense retrieval on those chunks, the central claim does not extend beyond the random-split assumption.","supporting_citations":[],"review_version":1}