{"id":"be5f631e-7633-4042-b0b4-fa0a7e55dc39","arxiv_id":"2411.10281","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multidimensional extension of Byte Pair Encoding compresses visual token grids losslessly into shorter sequences, improving transformer-based generation FID on image and 3D datasets.","lead":"This paper introduces a way to shorten the token sequences used by AI image generators by merging frequently repeated spatial patterns, similar to how text compression merges common word fragments. It reports that these shorter sequences make transformer-based image and 3D shape generation faster and more accurate, with only a small increase in preprocessing cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1 never updates a merged token's anchor point, so the paper's lossless claim and anchor-based sequence/positional encoding are not well-defined for non-convex tokens.","rationale":"Good-faith reading: the paper is a systems/empirical contribution; the core idea is to extend BPE to 2D/3D using constellation counts and anchors. For the central claim to hold, MDBPE must be a well-defined lossless preprocessing step and the resulting compressed sequence must be causally usable by a transformer. The weakest link is the anchor bookkeeping: the pseudocode creates a new merged token but never says what its anchor is or how expansion offsets are stored. The example above shows that the 'anchor' required by the definition can be n's old anchor even though the implementer is told to keep p's id and (implicitly) p's anchor. This makes both decoding and the token-order/positional encoding of Sec. 4.3 ambiguous. This is not a complaint that the method disagrees with prior consensus; it is an internal consistency gap. I agree with the reader that the empirical comparison is also under-controlled (no error bars, MDBPE package includes new positional encodings and codebook collapse), but that concern can be addressed by ablation; the anchor gap goes to the definition of the method. If the released code happens to update anchors implicitly, then the paper is simply missing a critical detail, and the fix plus a round-trip test would resolve the concern. Hence the verdict should remain conditional rather than reject: the contribution is plausible and potentially reproducible, but the conditions must include a precise anchor-update rule and a lossless reconstruction test, not only code and ablations.","tokens_in":17030,"tokens_out":22707,"duration_ms":238604,"concrete_test":"Implement Algorithm 1 exactly as written (store one anchor per unique id and do not update it in the replace step). Construct a small corpus of 3x3 grids that forces the merge sequence above: first merge the tokens at (2,0) and (2,1) into one vertical token, then merge that token with the token at (1,1). After the second merge, assert stored_anchor == row-major-first-occupied-pixel. Then expand each class according to its recorded (p.class, n.class, vpn) rule from the compressed sequence and compare to the input grid. If either check fails, the lossless claim is false for the algorithm as published. If the authors' code updates anchors implicitly, the discrepancy should be documented in Algorithm 1 and the round-trip test should pass.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.1 defines an anchor as the leftmost pixel in the uppermost row of a token, and Sec. 4.2 says the final sequence is written top-to-bottom, left-to-right at anchor positions. In Algorithm 1's replace step (lines 23-28), when a token n is merged into p, the code sets id[indn]=p.id and assigns a new class to both, but no anchor is updated. The merged token therefore keeps p's old anchor. This is inconsistent with the definition: a neighbor token n can contain a pixel above or to the left of p's anchor while still being adjacent to p. Concretely, build a corpus where (2,0) and (2,1) are first merged into a vertical token with anchor (2,0); then take a token p at (1,1) and the vertical token as its right neighbour via (2,1). The constellation vector is (1,1)-(2,0)=(-1,1). After the merge, the occupied set has row-major first pixel (2,0), not p's anchor (1,1). If the sequence is emitted at the stored anchor (1,1), the row-major 'first occurrence' rule in Sec. 4.2 is violated; if the sequence is instead emitted at the first occupied pixel, the expansion rule for the new class does not know where p and n were relative to that pixel. In either case the 'lossless preprocessing step' claim (Abstract, Sec. 4) is not established, and the next-token/positional encodings of Sec. 4.3, which depend on anchors, are ill-defined for exactly the non-convex tokens the paper says it supports. The text provides no proof, data structure, or invariant restoring anchor consistency after merges.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Multidimensional Byte Pair Encoding (MDBPE), a tokenization preprocessing step that extends BPE from 1D text to 2D image and 3D voxel grids. The algorithm counts frequent pairs of adjacent tokens together with their relative alignment (constellation vector), merges the most frequent constellation into a new token, and outputs a condensed sequence of token classes at anchor points. The paper also proposes two positional encoding modifications (next-token position and integrated spatial coverage) and a lossy codebook-collapse variant. Experiments on MNIST, CIFAR-10, SVHN, CelebA, and ShapeNet report improved FID, Coverage, and MMD for autoregressive transformers compared to vanilla transformers, pixel-level BPE, and the Octree Transformer.","tokens_in":17372,"tokens_out":12115,"duration_ms":111336,"significance":"MDBPE is a simple and potentially compelling add-on to existing visual tokenizers: it is agnostic to the base quantizer, is claimed to be lossless, and, according to Appendix A, increases preprocessing cost by only a constant factor. The authors provide code and evaluate across multiple datasets and modalities, which is a strength. If the losslessness and the attribution of the improvements are rigorously established, the method could be a broadly useful preprocessing technique for visual autoregressive models.","major_comments":[{"comment":"The paper claims that MDBPE is a 'lossless preprocessing step' (Abstract, Sec. 4), but it never defines the inverse mapping from a condensed token sequence back to the original grid. Algorithm 1 stores only token classes and unique IDs; it does not maintain a dictionary that maps each newly introduced token class to its spatial footprint (the set of covered pixels relative to the anchor). Without such a dictionary, a generated sequence cannot be rendered into an image, and the losslessness claim cannot be verified. Furthermore, the definition of an anchor as 'the leftmost pixel in the uppermost row' (Sec. 4.1) is stated as a property of a token, but Algorithm 1 does not update any stored anchor when two tokens are merged (lines 23-28). The paper should clarify whether the anchor is a stored field or recomputed on the fly; a merge of a token with anchor (1,1) and a vertical token with anchor (2,0) (adjacent via the pixel at (2,1)) yields a union whose row-major first pixel is (2,0), not the left constituent's anchor (1,1). This ambiguity directly affects the sequence-extraction rule in Sec. 4.2 and the next-token positional encoding in Sec. 4.3.","section":"Sec. 4.2 / Algorithm 1"},{"comment":"The comparison to Pixel-Level BPE (Razzhigaev et al.) is confounded. The MDBPE pipeline includes novel positional encodings introduced in Sec. 4.3 (Next Token Encoding and Integrated Positional Encoding), while the 1D BPE baseline appears not to use them. Since Fig. 7 shows that these encodings reduce test loss on their own, the FID gaps in Table 1 (e.g., 36.53 vs 33.93 on CIFAR-10 VQ-VAE) cannot be attributed solely to multidimensional compression. The authors should ablate the new encodings for both the 1D BPE baseline and MDBPE, or add a 1D BPE baseline with the same encodings, to isolate the effect of multidimensionality.","section":"Sec. 5.3 / Table 1 / Fig. 14"},{"comment":"All reported FID, Coverage, and MMD values are single-run point estimates without error bars or multiple seeds. Several improvements are small (e.g., CelebA VQ-VAE: 52.15 to 50.87 in Table 1; CIFAR-10 VQ-VAE: 36.53 to 33.93 in the same table), so the claimed gains might not be statistically significant. Please provide means and standard deviations over at least three independent training runs for the main comparisons.","section":"Tables 1, 2, 4, 5"},{"comment":"The training-speed boost is partly achieved by pruning the 5% of sequences with the longest length ('lastly, we further boost training speed by pruning the dataset from the 5% with the longest sequence length'). Removing the longest sequences changes the training set and reduces the effective padding length. The paper does not state whether the vanilla and 1D BPE baselines are trained on the same pruned data; if they are not, the comparisons in Table 5 and Fig. 10 are not controlled. The authors should either apply identical pruning to all methods or report the results without pruning.","section":"Sec. 4.4 / Table 5 / Fig. 10"}],"minor_comments":[{"comment":"The label 'Compression' is used with opposite conventions: in Table 1, lower values correspond to shorter sequences (e.g., 59% for MDBPE vs 65% for 1D BPE), but in Table 5 the header 'Compression↓' with 100% for the 'Full' codebook implies that lower is better as well. This should be clarified consistently, e.g., by stating explicitly that the numbers denote the remaining sequence length as a percentage of the original.","section":"Tables 1 and 5"},{"comment":"Several resolutions are typeset ambiguously as '282', '322', '162', '323'; these should be written as 28x28, 32x32, 16x16, and 32x32x32 for readability.","section":"Table 2 and Sec. 5.2"},{"comment":"The notation for positional encodings is inconsistent: the text uses 'positional encoding(i)' in the IPE formula, while Fig. 5 uses 'Pe' and 'P_e'. Use a single symbol throughout.","section":"Sec. 4.3, Fig. 5"},{"comment":"The paper states that the full codebase, including the faster C++ implementation, will be released, but only a small MNIST demo is currently linked. Please include the full code at submission time so that the reported results can be reproduced.","section":"Footnote 1"},{"comment":"The sentence 'VQ-VAEs are often regularised to avoid unused codebook entries' is followed by a list of citations, but the exact form of regularisation used in the experiments (e.g., codebook reset or commitment loss) is not described. Please specify the regularisation method in Appendix F.","section":"Sec. 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper has several strengths: it provides code, tests on multiple modalities, and reports consistent improvements over several baselines. However, the technical description lacks a precise specification of the inverse mapping (the shape dictionary), which is essential for the losslessness claim. The lack of error bars and the confounding of multidimensional BPE with new positional encodings are also substantive concerns. The editor may want to ask the authors to make the code and the full inverse procedure available before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The multidimensional extension of BPE is real and useful. The authors bring BPE from 1D to 2D/3D by counting token constellations with anchor points and offset vectors, then merging the most frequent pairs. They also add sensible auxiliary signals - next-token position, integrated positional encoding over the token shape, and a lossy codebook-collapse trick. That is a solid package, and the experiments back it up: across MNIST, CIFAR-10, SVHN, CelebA, and ShapeNet, compressed sequences consistently reach better FID than their uncompressed baselines, often with fewer FLOPs. The claim that this is a cheap add-on (about 2x preprocessing cost) also holds up. I believe the central hypothesis - shorter, content-adaptive sequences are easier for transformers to process - is supported by the evidence.\n\nWhere I would push back: the evaluation is single-run FID with no error bars or seeds. For a method whose whole pitch is consistent improvement, that is a genuine weakness; the differences are often small enough that variance could change the ranking. The ablations also conflate several innovations. Table 1 compares MDBPE to Pixel-level BPE, but MDBPE changes three things at once: multidimensional pairs, latent-space tokenization, and new positional encodings. You cannot attribute the gain to the multidimensionality from that table. The ablation in Fig. 7 isolates the positional encodings, which is good, but the core multidimensional-vs-1D comparison appears only in that mixed table.\n\nThe stress-test note about anchors is correct in spirit. Algorithm 1 never updates a merged token's anchor point after replacement. Under the paper's own definition of anchor (leftmost pixel of the uppermost row), a merge can leave the stored anchor inconsistent with that definition, and the sequence-extraction rule in Sec. 4.2 references the first occurrence in row-major order. This is a real ambiguity in the write-up. It is probably fixable by either updating anchors after each merge or explicitly defining the anchor as a fixed reference point that is part of the token, with shapes defined relative to it. The lossless claim needs a clear invariant, and this section should be rewritten. I do not think it sinks the method, because the representation can still be invertible if the token shape is defined relative to the stored anchor, but as written the description is internally inconsistent.\n\nThe code is not yet fully released, which is another condition I would attach. The MNIST demo is nice, but the paper promises a full codebase and a C++ implementation; without it, the reproducibility bar is not met.\n\nWho is this for? Anyone working on autoregressive visual generation or tokenization for multimodal LLMs. It is a worthwhile incremental contribution, not a paradigm shift. I would send it to serious peer review, with conditions: release the code, report error bars, and clarify the anchor update rule.","headline":"Useful generalization of BPE to 2D/3D token grids with consistent evidence of improved autoregressive generation; the core idea holds, but the paper needs to fix an anchor-consistency gap and report variance.","tokens_in":814,"tokens_out":801,"would_cite":true,"duration_ms":51190,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multidimensional byte pair encoding shortens visual token sequences and makes autoregressive transformers converge faster to better FID scores.","keywords":["byte pair encoding","visual tokenisation","autoregressive generation","sequence compression","image transformers","voxel grids","VQ-VAE","codebook collapse"],"falsifier":"Train the same transformer on sequences shortened by the least frequent token constellations instead of the most frequent ones, keeping sequence length and vocabulary size matched; if FID still improves over the uncompressed baseline, then shortness alone drives the effect and the frequency-based merge rule is not essential. Alternatively, match sequence length across two different token-shape vocabularies: if the FID gap persists, token shape carries part of the benefit.","tokens_in":16785,"feed_emoji":"🖼️","tokens_out":7515,"duration_ms":70820,"temperature":0.7,"pith_summary":"The paper argues that visual data generation improves when token sequences are shortened by a multidimensional extension of Byte Pair Encoding before a transformer ever sees them. The method, MDBPE, counts frequent neighbouring token constellations in horizontal and vertical directions, merges the most frequent constellation into a new token, and repeats until a vocabulary budget is reached; because the merge records anchor points and offset vectors, it is lossless and can form non-convex token shapes. The result is sequences that are shorter and whose information is more evenly distributed, condensing empty regions into single tokens. The paper reports that autoregressive transformers trained on these sequences converge earlier and reach better FID scores on several image datasets, and that the same recipe transfers to 3D voxel grids and signed-distance-function lattices. The cost of the extra dimension is a factor of two in preprocessing time compared with 1D BPE, which the paper argues is negligible relative to transformer training.","feed_headline":"Shorter token sequences improve image generation quality","feed_subtitle":"A lossless multidimensional byte-pair encoder condenses image tokens, cutting training time and improving FID.","key_machinery":"The load-bearing object is the constellation count: instead of counting adjacent token pairs along one axis as 1D BPE does, MDBPE slides a pairwise mask horizontally and vertically and counts triples of token class, neighbour class, and offset vector, where the anchor point of a token is the leftmost pixel in its uppermost row and the offset vector records the alignment between two anchors. Each pixel also carries a unique instance ID, so that when a constellation is chosen for merging, every pixel in both tokens can be re-labelled with a new token class and one shared ID; the final sequence is produced by scanning top-to-bottom, left-to-right and emitting each unique ID once at its anchor. This machinery is what lets the algorithm compress in two or three dimensions, form non-convex merged tokens, remain invertible, and keep the extra cost at roughly twice the pair count of 1D BPE.","core_discovery":"On the paper's own terms, the central discovery is that shortening visual token sequences by merging frequent constellations of neighbouring tokens makes them easier for transformers to process, not harder, even though the vocabulary grows. MDBPE is a lossless preprocessing add-on to existing discrete tokenisations: given a grid of VQ-VAE or VQGAN codes, greyscale values, or quantised colours, it repeatedly replaces the most frequent horizontally or vertically adjacent token pair with a fresh token, using anchor points and offset vectors to tell different alignments of the same pair apart. The extracted sequence writes each token once at its anchor position, so the sequence is shorter and each token can cover a larger area. Across MNIST, CIFAR-10, SVHN, CelebA, and ImageNet, the paper finds that transformers on these condensed sequences converge earlier, reach lower test loss and better FID, and can generate $32^{3}$ voxel grids that would not fit in consumer GPU memory without compression. A lossy extension that clusters the codebook into fewer representative tokens further amplifies compression, and extra positional encodings describing token area and next-token position improve learning.","pith_inferences":["The frequency-only merge rule is the main unproved choice; a controlled comparison against entropy-based or difficulty-weighted merge criteria would show whether the gains come from shortening itself or from the specific shapes frequency selection produces.","The paper's evidence is on standalone autoregressive transformers; whether the same shortening helps multimodal LLMs that consume visual tokens is an extension the authors suggest but do not test.","If the shortness effect is the active ingredient, a minimal test would be to hold sequence length fixed and vary only token shapes, or hold shapes fixed and vary sequence length, to separate the two factors.","The lossy codebook collapse suggests that posterior clustering of a trained discrete codebook can substitute for expensive retraining of a smaller-vocabulary VQGAN, a recipe that could transfer to other discrete generative pipelines."],"forward_implications":["Autoregressive image generation models trained on MDBPE-compressed sequences converge earlier and reach better final FID scores than the same transformers trained on uncompressed VQ-VAE, VQGAN, or colour-quantised sequences on MNIST, CIFAR-10, SVHN, and CelebA.","Because attention cost scales quadratically with sequence length, shorter sequences translate directly into lower FLOPs, lower memory, and larger batch sizes, so the same consumer GPU can train models on compressed sequences faster.","The 3D extension compresses voxel and SDF token grids strongly enough, for example to 47% of original length on 8^3 SDF grids, to bring 32^3 voxel autoregressive generation within consumer GPU memory, where uncompressed attention would not fit.","Lossy codebook collapse, snapping a VQGAN codebook of 2048 entries down to far fewer representatives, roughly preserves visual fidelity while shortening sequences further, so it can be used as a cheap speed-up for prototyping and training.","Diversity is not sacrificed: the paper reports train and test losses that stay close even under strong compression, and nearest-neighbour checks on CelebA show the model is not simply memorising training images."],"supporting_citations":[{"why":"It supplies the 1D pixel-level BPE baseline on quantised images that MDBPE is compared against.","marker":"[50]"},{"why":"It supplies the VQGAN transformer training recipe whose time and FID MDBPE is compared with.","marker":"[19]"},{"why":"It supplies the VQ-VAE discrete latent token grids that MDBPE compresses in most image experiments.","marker":"[61]"},{"why":"It supplies the adaptive octree-based 3D generation baseline for the 32^3 voxel comparisons.","marker":"[27]"},{"why":"It supplies the original byte pair encoding text compression scheme that MDBPE generalises to multiple dimensions.","marker":"[56]"},{"why":"It supplies the integrated positional encoding idea reused to encode token shape coverage.","marker":"[5]"},{"why":"It defines the FID metric used to measure generation quality throughout the evaluation.","marker":"[23]"},{"why":"It supplies the farthest point sampling used to initialise the codebook collapse.","marker":"[44]"}],"fun_headline_variants":["Multidimensional byte-pair encoding condenses visual tokens","Lossless token compression boosts image transformer training","Shorter visual sequences from multidimensional byte pair encoding","MDBPE: Merge frequent token pairs for faster image generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's main load-bearing premise is that merging the most frequent token constellations, chosen by global training-set frequency counts, leaves exactly the information a transformer needs for generation; this frequency heuristic is empirical, and the claim that anchor-point extraction stays losslessly invertible for arbitrary non-convex token shapes is asserted rather than proved.","fun_headline_variants_meta":{"raw":{"variants":["Multidimensional byte-pair encoding condenses visual tokens","Lossless token compression boosts image transformer training","Shorter visual sequences from multidimensional byte pair encoding","MDBPE: Merge frequent token pairs for faster image generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1280,"prompt_tokens":978,"completion_tokens":302,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":240}},"tokens_in":594,"tokens_out":302,"duration_ms":3474,"temperature":1.0,"reasoning_tokens":240,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:46:56.719241+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same transformer on sequences shortened by the least frequent token constellations instead of the most frequent ones, keeping sequence length and vocabulary size matched; if FID still improves over the uncompressed baseline, then shortness alone drives the effect and the frequency-based merge rule is not essential. Alternatively, match sequence length across two different token-shape vocabularies: if the FID gap persists, token shape carries part of the benefit.","supporting_citations":[{"cited_title":"Pixel-level BPE for auto-regressive image generation","cited_arxiv_id":null,"evidence_quote":"It supplies the 1D pixel-level BPE baseline on quantised images that MDBPE is compared against."},{"cited_title":"Octree trans- former: Autoregressive 3d shape generation on hierarchi- cally structured sequences","cited_arxiv_id":null,"evidence_quote":"It supplies the adaptive octree-based 3D generation baseline for the 32^3 voxel comparisons."},{"cited_title":"Byte pair encoding: A text compression scheme that accelerates pattern matching","cited_arxiv_id":null,"evidence_quote":"It supplies the original byte pair encoding text compression scheme that MDBPE generalises to multiple dimensions."}],"review_version":1}