{"id":"4900a1ca-9e29-407b-89b5-47bf54657f2f","arxiv_id":"2412.06926","paper_version":5,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Using a shortest-path search to split text with the same token vocabulary saves 3-5% of tokens on many languages, but downstream accuracy gains are mixed and confounded by the experimental design.","lead":"This paper swaps the standard left-to-right 'greedy' token splitting used by OpenAI's tokenizers for a dynamic-programming search that always picks the fewest tokens from the same vocabulary. The authors report 3-5% token savings across many languages and some accuracy gains on smaller GPT-2 models, though the evidence is uneven and partly circular.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Downstream gains are unproven: the models were fine-tuned from a greedy-pretrained checkpoint, so optimal-segmentation accuracy improvements may reflect token-distribution shift rather than better segmentation.","rationale":"A good-faith reading shows the paper has a correct but elementary dynamic-programming segmentation algorithm; the intrinsic token-count reduction is mathematically guaranteed by the definition of S*, so the 3-5% TSR results are not a substantive empirical discovery. The only non-tautological contribution is the extrinsic claim that optimal segmentation improves downstream accuracy. That claim is tested by starting from a GPT-2 checkpoint pretrained with greedy tokenization and fine-tuning it with optimal tokenization. The prior distribution of token IDs is different, so any observed accuracy change could be explained by distributional shift rather than by the segmentation being better. The assumption in Section 5.3 that the greedy-pretrained model 'only has to learn the difference in the distribution of tokens' is not self-evident and is not supported by any experiment. The reader's weakest_assumption identifies exactly this issue, and I agree. The decisive check is a from-scratch pretraining comparison; until that is run, the central accuracy claim is not established. I would leave the reader's REJECT verdict unchanged.","tokens_in":17643,"tokens_out":3280,"duration_ms":37620,"concrete_test":"Pretrain two GPT-2 120M models from scratch on the same OpenWebText corpus, same vocabulary, same hyperparameters, and at least three random seeds, one using greedy segmentation and one using optimal segmentation at every training step. Then fine-tune both models on the same downstream benchmarks in Table 7 (Emot, WreTe, XNLI, TyDiQA-GoldP, QQP, Story Cloze) and compare mean accuracy with standard deviations. If optimal segmentation does not consistently beat greedy across these tasks, the reported gains are an artifact of fine-tuning from a greedy-pretrained checkpoint.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claim ('up to a 10% increase in accuracy', Abstract; Table 7) depends on an untested adaptation assumption. Section 5.3 states: 'We did not do a complete pretraining from scratch as the model pre-trained with greedy segmentation only has to learn the difference in the distribution of tokens with optimal segmentation.' This is asserted, not argued. A GPT-2 checkpoint was pretrained on sequences produced by greedy segmentation, so its token embeddings and attention patterns are adapted to that distribution. When the same text is re-tokenized with optimal segmentation, the model receives different token IDs, different sequence lengths, and different co-occurrence structure. Fine-tuning on a downstream task can adapt to this distribution shift, but the resulting accuracy differences are not identifiable with the causal effect of optimal segmentation. The paper's only direct evidence for the headline extrinsic benefit is this fine-tuning comparison, making the uncontrolled pretraining regime the load-bearing part of the contribution. The English QQP result (greedy better in Table 7) and the inconsistency between Tables 5 and 7 for Turkish XNLI (100% non-zero TSR, yet All and TSR* accuracies differ) further weaken the pattern, but even without those issues the central comparison is confounded.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes replacing the greedy segmentation step of BPE tokenizers with an optimal segmentation that minimizes the number of tokens for a given document, using a dynamic programming algorithm over a reversed-trie of the fixed vocabulary. The authors evaluate the token-saving ratio (TSR) across 116 languages on CC-100 and three OpenAI tokenizer vocabularies, and then fine-tune GPT-2 (120M and 350M) checkpoints on OpenWebText with either greedy or optimal tokenization, evaluating downstream tasks in English, Finnish, Indonesian, and Turkish. They report 3–5% average token savings, larger savings for long words, and accuracy improvements up to about 10% on non-English tasks.","tokens_in":17827,"tokens_out":3581,"duration_ms":37195,"significance":"If the downstream claims held, the work would have practical value: it proposes a drop-in replacement for the segmentation stage that requires no vocabulary change or retraining. The algorithm is clearly described and its token-count minimization is mathematically sound and verifiable. The intrinsic measurement of token savings across languages is a useful empirical contribution, especially the word-length correlation. However, the paper's headline claim of accuracy improvements rests on a confounded experimental setup, and the reported results contain an inconsistency that undermines confidence in the extrinsic evaluation. The central contribution is therefore not yet established.","major_comments":[{"comment":"The extrinsic evaluation is confounded by distribution shift between pretraining and fine-tuning. The models are initialized from GPT-2 checkpoints pretrained on greedy tokenization and then fine-tuned on OpenWebText with optimal tokenization. The authors justify this by asserting that the greedy-pretrained model 'only has to learn the difference in the distribution of tokens with optimal segmentation' (Section 5.3), but no argument or supporting experiment is provided. Under optimal segmentation, the model sees different token IDs, different sequence lengths, and different co-occurrence statistics; the fine-tuned accuracy differences in Table 7 therefore conflate the effect of segmentation with the model's ability to adapt to a new token distribution. Without pretraining from scratch (or at least continued pretraining on optimal-tokenized data before task fine-tuning), the 'up to a 10% increase in accuracy' claim in the Abstract is not supported.","section":"Section 5.3 and Table 7"},{"comment":"There is a direct inconsistency between Table 5 and Table 7 for Turkish XNLI. Table 5 reports that 100% of Turkish XNLI samples have non-zero TSR, which means the 'All' and 'TSR*' subsets are identical. However, Table 7 lists different accuracies for 'All' and 'TSR*' in the Turkish XNLI rows (e.g., 120M Greedy: 64.35 vs 63.83; 120M Optimal: 64.91 vs 64.59). Either the TSR* subset construction is incorrect, or Table 7 contains reporting errors. This discrepancy affects a reported result central to the paper's conclusion and must be resolved.","section":"Table 5 vs Table 7"},{"comment":"The headline 'optimal segmentation reduces token count' is true by construction: Eq. (2) defines S* as the minimizer of |S(d)| over a fixed vocabulary, so comparing S* to any other feasible segmentation (e.g., greedy) yields a non-negative TSR by definition. The paper's framing of this as an 'improvement' is therefore partially circular. The empirical content lies in the magnitude of the savings across languages and vocabularies, and the paper would be more accurately framed as measuring that magnitude rather than as demonstrating that optimal segmentation beats greedy in principle. The abstract and introduction should be revised to avoid presenting a definitional property as a discovered result.","section":"Section 3.2, Eq. (1) and Section 4.1, Eq. (2)"},{"comment":"The TSR* subset analysis is post-hoc and lacks statistical controls. Restricting to examples where greedy and optimal differ (Table 5) can amplify small differences through selection effects, yet the paper reports no error bars, no number of random seeds, and no significance tests. For example, the English Story Cloze 350M result shows a 7.83% gain in TSR* versus a 0.43% gain on the full dataset, which is a large swing on a very small subset (6.15% of the data, hence about 13 examples if the dataset has ~200 examples). Without confidence intervals or multiple runs, these differences may reflect noise rather than a systematic effect. The authors should either provide repeated fine-tuning runs with variance estimates or temper their conclusions.","section":"Section 5.2 and Table 7"}],"minor_comments":[{"comment":"The caption of Table 2 defines 'TSR' as 'Token Stability Ratio', but Section 3.2 defines TSR as 'Token Saving Ratio'. This inconsistency should be corrected.","section":"Table 2 caption"},{"comment":"The pseudocode initializes dp[n] to 0 and uses dp[j-1] for j=0; the base case dp[-1]=0 is described in the text but not clearly represented in the code. Clarify the indexing (e.g., use 1-indexed or explicitly define dp[-1]).","section":"Algorithm 1"},{"comment":"The proof of optimality contains typos and confusing index notation, such as 'dp[i] = dp[km-2] + 1' where the recurrence should refer to dp[km-2] as a candidate minimum, not an equality, and the line 'dp[i] = dp[ki] + m - 1 - i' appears to have off-by-one errors. The proof should be rewritten for clarity.","section":"Appendix B"},{"comment":"The description of the fine-tuning setup is insufficient for reproducibility: no learning rate, batch size, number of epochs, or random seed is reported. Appendix A lists only model architecture dimensions, not training hyperparameters.","section":"Section 5.3"},{"comment":"The claim that word length 'has a strong correlation' with TSR is supported only by a figure (Figure 1); no correlation coefficient or confidence interval is reported. A quantitative correlation statistic would strengthen the claim.","section":"Section 6.1.2"}],"recommendation":"reject","confidential_remarks":"The core problem is the confounded extrinsic evaluation, which is not fixable by textual revisions; retraining from scratch would be required to support the main accuracy claim. The Table 5/7 inconsistency further signals that the reported numbers may not be reliable. The intrinsic token-saving measurements are sound but are partially definitional and likely insufficient for a standalone publication of this scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the intrinsic token-saving catalogue is the real contribution; the downstream accuracy claims are not supported by the experiments as run.\n\nWhat's genuinely useful: the paper defines TSR cleanly, implements optimal minimum-token segmentation over a fixed BPE vocabulary via DP plus reversed trie, and measures TSR across 116 languages. The headline 3-5% savings, with larger gains on long/rare words, is an honest empirical finding that others can reproduce with the public tokenizers. The qualitative examples in Table 2 are illustrative. Those parts are worth having.\n\nThe soft spots are where the paper reaches beyond the intrinsic result. First, the token-saving direction is guaranteed by construction: TSR compares the minimizer against a greedy feasible solution, so positive TSR is a theorem, not an experimental discovery. The magnitude is empirical, fine, but the framing oversells it. Second, and more seriously, the extrinsic evaluation is confounded. Section 5.3 says they did not pretrain from scratch because a greedy-pretrained model 'only has to learn the difference in the distribution of tokens.' That is an assumption, not an argument. Starting from a checkpoint trained on greedy-token sequences and fine-tuning on optimal-token sequences conflates tokenization quality with distribution shift. A controlled pretraining comparison, or at least continued pretraining on both token distributions, is needed to support the 'up to 10% accuracy' claim. Third, there are no error bars or seeds, and the TSR* subset is selected post hoc after seeing which examples differ, which can inflate gains. Fourth, Table 5 and Table 7 conflict: Turkish XNLI is listed at 100% non-zero TSR, so the All and TSR* columns should be identical, but they differ by about half a point. That is a direct internal inconsistency, not a matter of interpretation.\n\nI would not take the extrinsic numbers at face value. But the paper is not empty: the intrinsic catalogue has independent value, and the authors clearly know the adjacent work (Uzan et al., Schmidt et al., Goldman et al.). If the extrinsic experiments were redone with proper controls and full reporting, a revised version could be a decent venue piece. As is, it should not be accepted on the current evidence.\n\nRecommendation: send to peer review with a request for major revision; a good referee will catch the pretraining confound and the table inconsistency. The intrinsic section alone does not justify acceptance, but the paper deserves the referee time.","headline":"The intrinsic token-saving catalogue is solid and reusable, but the downstream accuracy claims rest on a confounded fine-tuning setup and a table inconsistency.","tokens_in":18406,"tokens_out":2888,"would_cite":true,"duration_ms":28623,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Optimal token splits cut token counts 3-5%, lift accuracy up to 10%.","keywords":["optimal segmentation","BPE tokenization","token saving ratio","low-resource languages","GPT-2","dynamic programming","morphological complexity","tokenization efficiency"],"falsifier":"Train two identical GPT-2-size models from scratch on the same corpus and vocabulary—one with greedy BPE segmentation and one with optimal (minimum-token) segmentation—and compare their perplexity and task accuracy. If the optimally segmented model does not match or beat the greedy model after equivalent training, the paper's claim that optimal segmentation alone improves performance would be refuted.","tokens_in":17401,"feed_emoji":"🧩","tokens_out":5052,"duration_ms":45218,"temperature":0.7,"pith_summary":"The paper claims that the standard greedy segmentation used by BPE tokenizers is suboptimal, and that replacing it with a minimum-token segmentation of the same vocabulary—computed by a Viterbi-style dynamic programming pass over a reversed-token trie—cuts token counts by 3-5% and improves downstream accuracy by up to 10% on low-resource and morphologically rich languages. The point is that a deployment-ready tokenizer can benefit from shorter sequences and better task performance without retraining a new vocabulary or pretraining from scratch. This matters because token bloat inflates compute costs quadratically with sequence length and disproportionately harms low-resource languages.","feed_headline":"Optimal token splits cut token counts 3-5%, lift accuracy up to 10%","feed_subtitle":"A dynamic-programming tokenizer beats greedy BPE for low-resource languages without changing the vocabulary.","key_machinery":"The central object is the dynamic-programming recurrence $dp[i] = \\min_{0 \\le j \\le i,\\ d[j..i] \\in V} (dp[j-1] + 1)$, which finds the minimum number of tokens to cover a document prefix. It is implemented with a trie built on reversed vocabulary tokens so membership tests run in $O(M)$ per position, where $M$ is the longest token length; the whole algorithm runs in $O(N \\cdot M)$, the same worst-case bound as greedy segmentation. The paper also introduces the Token Saving Ratio (TSR), defined as $(|S_B(d)| - |S_A(d)|)/|S_B(d)|$, to quantify compression gains between two segmentations.","core_discovery":"The core claim, stated on the paper's own terms, is that optimal segmentation $S^* = \\arg\\min_S |S(d)|$ over a fixed vocabulary $V$ is not only well-defined but practically achievable at the same worst-case time complexity as greedy BPE inference, and that the resulting sequences are both shorter and linguistically more coherent. The paper demonstrates this empirically across 116 languages using OpenAI tokenizers with 50K, 100K, and 200K vocabularies, reporting 3-5% average token savings and up to 20% savings on rare and long words, plus accuracy gains of up to 10% on classification and generation tasks in Finnish, Indonesian, and Turkish while keeping English perplexity essentially unchanged.","pith_inferences":["If the fine-tuning proxy holds, then any subword tokenizer (WordPiece, Unigram, SentencePiece, or future ones) could adopt the same min-token decoding to gain compression and accuracy, since the segmentation stage is decoupled from vocabulary construction.","The TSR* subset analysis implies that gains concentrate on morphologically complex tokens; a testable extension is to predict per-sample benefit from word-length and morphological features before deciding whether to use optimal or greedy decoding.","A direct pretraining-from-scratch comparison—optimal vs greedy segmentation on identical corpora and vocabularies—would settle whether the reported accuracy gains reflect true representation quality or merely adaptation to a shifted token distribution; the paper did not run it.","Combining optimal segmentation with vocabulary construction that is aware of the min-token objective could push compression beyond the 3-5% reported here, since the vocabulary is held fixed in the paper."],"forward_implications":["Replacing greedy decoding with the minimum-token segmentation of the same vocabulary yields 3-5% fewer tokens on average across 116 languages, with up to 20% compression on rare and long words.","On downstream classification and generation tasks, models using optimal segmentation show accuracy gains of up to 10%, with larger gains on the subset of examples where the two segmentations actually differ.","English-language perplexity stays essentially flat, so the gains on non-English languages do not come at the cost of English modeling quality.","Because the optimal segmentation algorithm has the same worst-case time complexity as greedy, it can be dropped into existing BPE-based tokenizers without changing the vocabulary or the model architecture.","Longer words and agglutinative morphologies (Finnish, Turkish, Indonesian) show the strongest token savings, suggesting the method targets exactly the languages where token bloat is worst."],"supporting_citations":[{"why":"Supplies the GPT-2 tokenizer and model family used as the greedy baseline and for fine-tuning.","marker":"Radford et al. 2019"},{"why":"Gives the Viterbi dynamic programming formulation on which the optimal segmentation algorithm is based.","marker":"Forney 1973"},{"why":"Provides the intrinsic/extrinsic evaluation methodology and evidence that compression correlates with downstream performance.","marker":"Goldman et al. 2024"},{"why":"Evaluates greedy tokenizer inference methods, framing the baseline the paper improves on.","marker":"Uzan et al. 2024"},{"why":"Provides the CC-100 multilingual dataset used for intrinsic measurement across 116 languages.","marker":"Wenzek et al. 2020"},{"why":"Documents tokenizer-induced unfairness between languages, motivating the low-resource focus.","marker":"Petrov et al. 2023"},{"why":"Quantifies the economic cost of tokenization disparities, motivating why token savings matter.","marker":"Ahia et al. 2023"}],"fun_headline_variants":["Optimal token splits cut tokens 3-5%, lift low-resource accuracy 10%","Optimal segmentation beats greedy BPE: 3-5% token savings, 10% accuracy lift","Optimal tokenization saves 3-5% tokens, boosts low-resource accuracy 10%","Dynamic programming tokenizer beats greedy BPE across 116 languages","Up to 20% fewer tokens on rare words, 10% accuracy boost via optimal splits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Fine-tuning a model that was pretrained with greedy tokenization on optimally tokenized text is a valid substitute for pretraining with optimal tokenization from the start; if that substitution fails, the reported accuracy gains could be artifacts of distribution shift rather than benefits of optimal segmentation.","fun_headline_variants_meta":{"raw":{"variants":["Optimal token splits cut tokens 3-5%, lift low-resource accuracy 10%","Optimal segmentation beats greedy BPE: 3-5% token savings, 10% accuracy lift","Optimal tokenization saves 3-5% tokens, boosts low-resource accuracy 10%","Dynamic programming tokenizer beats greedy BPE across 116 languages","Up to 20% fewer tokens on rare words, 10% accuracy boost via optimal splits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00307,"raw_usage":{"total_tokens":11564,"prompt_tokens":821,"completion_tokens":10743,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":437,"completion_tokens_details":{"reasoning_tokens":10626}},"tokens_in":437,"tokens_out":10743,"duration_ms":72672,"temperature":1.0,"reasoning_tokens":10626,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:20:31.266295+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two identical GPT-2-size models from scratch on the same corpus and vocabulary—one with greedy BPE segmentation and one with optimal (minimum-token) segmentation—and compare their perplexity and task accuracy. If the optimally segmented model does not match or beat the greedy model after equivalent training, the paper's claim that optimal segmentation alone improves performance would be refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GPT-2 tokenizer and model family used as the greedy baseline and for fine-tuning."}],"review_version":1}