{"id":"f5c0f863-c29c-4ac6-8d44-e80125b62e47","arxiv_id":"2412.15195","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"OptVQ uses Sinkhorn optimal transport instead of nearest-neighbor search for vector quantization, achieving full codebook utilization and improved image reconstruction.","lead":"A new vector quantization method, OptVQ, replaces the nearest-neighbor code lookup with an optimal transport assignment computed by the Sinkhorn algorithm, which balances code usage globally. It reports 100% codebook utilization and better image reconstruction than prior VQ models, promising more stable training for token-based generation models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Sinkhorn OT in Eq. (8) imposes equal row and column marginals, which is infeasible when the number of features l differs from the codebook size n; this occurs in Table 3 (and likely MNIST/CIFAR), so the claimed balanced-assignment mechanism is not actually being optimized.","rationale":"The reader's weakest assumption is exactly the feasibility of the balanced OT problem when l != n, and I agree this is the most load-bearing formal gap. The main ImageNet comparison (n = 16,384, l = 16,384) is feasible, so the headline SOTA claim is not directly falsified by this issue, but Table 3, which provides the controlled evidence that OT alone improves utilization and reconstruction, is dominated by infeasible settings. The authors should either state that they are intentionally solving an unbalanced OT variant, or adjust the marginals so the problem is well-posed. The proposed concrete test would settle whether the infeasible implementation is responsible for the claimed gains. I also note the capacity mismatch from multi-head quantization in Table 1, but that is a comparison-design flaw rather than an internal inconsistency in the method; the infeasibility concern is more fundamental. Since the reader already requested clarification of the marginal matching issue, my stress-test does not change the verdict: the paper remains conditionally acceptable pending a feasible formulation or explicit justification of the unbalanced Sinkhorn procedure.","tokens_in":18516,"tokens_out":7369,"duration_ms":77923,"concrete_test":"Modify the Sinkhorn implementation in the released code to use feasible column marginals of l/n instead of 1 (so that total mass matches the row marginals), and re-run the Table 3 ablation for n = 128, 1024, and 4096 with l = 16,384. Record the codebook utilization and reconstruction loss. If utilization drops below 100% or the reconstruction loss moves back toward the VQ-VAE baseline, the reported Table 3 gains are an artifact of enforcing an infeasible column constraint rather than a genuine benefit of optimal transport.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The OT problem in Eq. (8) constrains A1_r = 1_r and A^T 1_c = 1_c. Summing the row constraint gives total mass l, while summing the column constraint gives total mass n; a doubly stochastic matrix with these marginals exists only if l = n. In Table 3, the latent resolution is 16x16 and the total batch size is 64, so l = 64 x 256 = 16,384 for every row, but the codebook size n ranges from 128 to 16,384. For n < 16,384 the constraint set is empty, so the stated problem has no solution. The Sinkhorn algorithm with alternating row and column normalization cannot converge to a doubly stochastic matrix in this infeasible case; after the final column normalization the column sums are 1 but row sums become l/n, not 1. The paper's theoretical motivation, that the equality constraints and entropy term 'ensure that each code and feature participate fully' in the assignment, does not apply to these settings. The main ImageNet result with n = 16,384 is feasible, but the controlled ablations in Table 3 that isolate the OT mechanism from capacity effects are run in the infeasible regime. Their 100% utilization and reconstruction gains therefore cannot be attributed to the stated balanced OT formulation. A secondary but related concern is that Table 1 uses B = 4 heads, giving an effective codebook of n^B, which is not capacity-matched to single-head baselines; however, the marginal infeasibility is the more fundamental formal gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OptVQ, a vector quantization method that replaces the conventional nearest-neighbor search with a Sinkhorn-based optimal transport assignment, together with a distance normalization strategy and a multi-head quantizer. The authors argue that nearest-neighbor quantization is prone to local minima and index collapse, and that a globally informed assignment through optimal transport avoids this. They report 100% codebook utilization and improved reconstruction quality over existing VQNs on MNIST, CIFAR-10, and ImageNet, and they release code for reproducibility.","tokens_in":18882,"tokens_out":7756,"duration_ms":66970,"significance":"If the central claims hold, OptVQ would be a practically useful plug-and-play quantizer: it could stabilize VQN training without specialized initialization or distillation, and its 100% codebook utilization together with better reconstruction would address a well-known failure mode. The paper includes a clear motivation, an algorithmic proposal, and a substantial set of experiments, including ablation studies and training statistics. The code release is a further strength. However, the formal optimal-transport problem in Eq. (8) is infeasible in most of the reported experimental settings, and the main comparison in Table 1 is not capacity-matched. These issues affect the interpretation of the central claims, so the current evidence is not yet fully convincing.","major_comments":[{"comment":"Equation (8) imposes A1_r=1_r and A^T1_c=1_c. Summing these constraints gives total mass l and n respectively, so a feasible A exists only when l=n. The paper never states how l and n are matched. In the ImageNet ablations of Table 3, the default settings (batch size 64, latent 16×16) give l=64×256=16,384 per assignment step, while n ranges from 128 to 16,384; for n<16,384 the constraint set is empty. The sentence in Sec. 3.2 that the constraints 'ensure that each code and feature participate fully' is therefore not supported in the regime where the controlled codebook-size ablations are run. Please either define the actual optimization problem being solved (e.g., an unbalanced OT formulation) or provide an analysis of why 5 Sinkhorn iterations on the infeasible problem produce the desired balanced assignment.","section":"Sec. 3.2, Eq. (8)"},{"comment":"The 100% codebook utilization reported in Table 3 is partly enforced by construction: the Sinkhorn column normalization in Eqs. (11)-(12) forces every code to receive equal total column mass, so the utilization measure is not an independent empirical outcome. To support the claim that OT improves utilization beyond the balancing mechanism, the paper should compare against an alternative assignment that uses only column normalization (or a uniform-prior soft assignment) without the full OT objective, and should report utilization for matched-capacity settings.","section":"Sec. 4.4, Table 3"},{"comment":"The state-of-the-art comparison in Table 1 is not capacity-matched: OptVQ uses B=4 heads with codebook size 16,384 per head, giving an effective codebook of size n^B, whereas most baselines use a single codebook. The reconstruction gains could therefore be attributed to the larger effective codebook rather than to the optimal-transport assignment. Please add an ablation with a single-head OptVQ at comparable effective capacity, or a multi-head baseline with the same B and codebook size, to isolate the effect of the OT mechanism.","section":"Table 1"},{"comment":"The proposed normalization replaces the original distance matrix D by D'', a monotone but nonlinear transformation. Because the Sinkhorn soft assignment depends on the magnitudes of the exponentiated costs, solving Eq. (8) with D'' is not equivalent to solving it with D. The paper should justify that this transformed cost preserves the intended global assignment properties, or treat the normalized objective as the actual method and analyze it directly.","section":"Sec. 3.3, Eqs. (11)-(12)"},{"comment":"The convergence claim that 'no more than 5 iterations are enough' is demonstrated only for a 10×10 instance (l=n=10). In the actual experiments l=16,384 and n varies over several orders of magnitude; with l≠n the alternating row/column normalization does not converge to a doubly stochastic matrix. The paper should either provide convergence experiments at the operating dimensions or restrict the claim to the feasible setting.","section":"Fig. 9 and Sec. 3.3"}],"minor_comments":[{"comment":"There are numerous typos and grammatical errors, e.g., 'develope' (Sec. 1), 'ourperforms' (Sec. 4.2), 'substatiating' (Sec. 5), 'obstables' (Sec. 5), and 'pionts' (Fig. 7 caption). A careful proofread is needed.","section":"Throughout"},{"comment":"The 'From Scratch' column uses the symbols \" and % without a legend; please clarify what these denote and how they were determined for each baseline.","section":"Table 1"},{"comment":"The consistency check would be more informative if it included a quantitative measure of agreement between OptVQ and nearest-neighbor assignments rather than only arrows in a 2D example.","section":"Sec. 4.3, Fig. 7"},{"comment":"The statement that 'the majority of codes are selected between 300 to 600 times' is not supported by the histogram in the figure, which appears to show a broader range; please either adjust the description or report the relevant quantiles.","section":"Appendix C, Fig. 10"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising empirical direction and releases code, which is valuable for the community. However, the formal gap in Eq. (8) is central: the stated OT problem is infeasible in most experimental configurations, so the theoretical motivation does not match the reported method. The '100% codebook utilization' should also be presented as a design feature of the column normalization rather than an emergent discovery. If the authors can reframe the method as an unbalanced or heuristic Sinkhorn assignment and add capacity-matched comparisons, the contribution could be solid. The current version, however, requires substantial revision before the central claims are supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about this paper. The core idea—replacing nearest-neighbor code lookup with a Sinkhorn-balanced assignment—is a genuinely new combination in the VQ tokenizer subfield, and the paper shows it works well in practice. But the formal statement of the optimal transport problem is incorrect for most of the experiments, and the headline ImageNet comparison is not capacity-matched. Both issues are fixable; this is not a rejected idea, it's a draft that needs a serious referee.\n\nWhat's actually new: using OT as the quantizer assignment in a VQ autoencoder, plus a normalization trick to stabilize Sinkhorn. The paper identifies a real problem (index collapse due to greedy nearest-neighbor assignment) and offers a plausible, simple fix. On MNIST and CIFAR they report codebook utilization jumping from under 1% to 100%, and the training dynamics visualizations are informative. The code is public.\n\nNow the soft spots, in proportion. The reader's stress-test is correct: Eq. (8) enforces both row and column marginal constraints, which is feasible only when the number of features l equals the codebook size n. In Table 3, l is fixed by batch size and latent resolution (64 x 256 = 16,384) while n ranges from 128 to 16,384; for n < 16,384 the constraint set is empty. The Sinkhorn iterations cannot produce a doubly stochastic matrix then; after the final column normalization, the row sums are l/n, not 1. What the algorithm actually does is enforce balanced column assignment—arguably what you want for codebook utilization—but it is not the balanced OT problem stated, and the paper doesn't acknowledge this. This is a theory and presentation gap, not necessarily a fatal flaw: the empirical results can still be explained by the column-balancing mechanism. Still, the authors should reformulate the objective or justify the infeasible iteration.\n\nThe second issue is Table 1. OptVQ uses 4 heads, giving an effective codebook of n^4, while most baselines are single-head. That alone could explain part of the reconstruction gains. The authors should compare against a single-head model with a similarly large codebook, or report the rate-distortion trade-off.\n\nThe paper is for anyone working on VQ tokenizers for image generation or discrete representation learning. It has a useful empirical core, but the theory needs cleanup.\n\nRecommendation: send to peer review conditionally. Ask the authors to fix the OT feasibility claim and add capacity-matched comparisons. The idea deserves referee time.","headline":"A fresh idea in VQ—Sinkhorn-based assignment—with real empirical promise, but the OT formulation is formally wrong in most experiments and the SOTA claim isn't capacity-matched; fixable, so worth refereeing.","tokens_in":19390,"tokens_out":4321,"would_cite":true,"duration_ms":30035,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T45"],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing the nearest-neighbor search in vector quantization with a Sinkhorn-based optimal transport assignment lets VQNs train stably from scratch, reach 100% codebook utilization, and beat current quantizers on reconstruction quality.","keywords":["vector quantization","optimal transport","Sinkhorn algorithm","codebook utilization","index collapse","image reconstruction","VQGAN","discrete tokenization"],"falsifier":"Run OptVQ on a batch where the number of features $l$ is much larger than the codebook size $n$, record the column sums of the assignment matrix after five Sinkhorn iterations, and count how many codes receive zero argmax picks; if the column sums are far from uniform or any code is never selected, the claimed 100% utilization is not produced by the mechanism the paper describes.","tokens_in":18275,"feed_emoji":"💯","tokens_out":9457,"duration_ms":66598,"temperature":0.7,"pith_summary":"Vector-quantized networks assign continuous features to discrete codebook entries by nearest-neighbor search, a greedy rule that routinely strands most entries unused (index collapse) and makes training depend on careful initialization and distillation. This paper argues that the collapse is a local-minimum artifact of the assignment rule itself, not a defect of the codebook, and replaces the search with a global optimal-transport assignment solved by the Sinkhorn algorithm. The resulting quantizer, OptVQ, reaches 100% codebook utilization across codebook sizes and latent dimensions, and in image-reconstruction experiments it improves PSNR, SSIM, LPIPS, and rFID over current vector-quantized models. If the diagnosis is right, a simple swap of the assignment step removes the main source of VQ training instability.","feed_headline":"Swap nearest-neighbor for Sinkhorn: 100% codebook use","feed_subtitle":"OptVQ treats token assignment as optimal transport, escaping the local-minima traps that strand VQ codebooks.","key_machinery":"The load-bearing object is the assignment matrix $A \\in R^{l \\times n}_+$ defined by the entropy-regularized optimal transport problem $\\min_A \\mathrm{Tr}(A^T D) - \\frac{1}{\\epsilon} H(A)$ with row and column sum constraints $A\\mathbf{1}_r = \\mathbf{1}_r$ and $A^T \\mathbf{1}_c = \\mathbf{1}_c$, where $D_{ij}$ is the distance between feature $z_i$ and code $c_j$. Instead of the nearest neighbor, the quantizer sends each feature to the code with the largest entry in $A$. The Sinkhorn-Knopp iterations, which alternate row and column normalization starting from $A^0 = e^{-\\epsilon D}$, compute this matrix in about five iterations, and a normalization of $D$ decouples the choice of $\\epsilon$ from the scale of the data. Because the row and column sums force every code to participate, the argmax over $A$ yields balanced codebook usage.","core_discovery":"The paper's central claim is that the index-collapse failure of vector-quantized networks is caused by the local, greedy nature of nearest-neighbor assignment: once a feature lies inside a code vector's Voronoi cell, the commitment loss keeps pulling it back into that same cell, so badly initialized codes are never repaired. OptVQ replaces the nearest-neighbor rule with a Sinkhorn-based optimal-transport assignment, solving for a soft assignment matrix $A$ that both respects distances and spreads mass over all codes and all features, then choosing the code with the largest entry per feature. The authors report that this removes the need for special initialization and distillation, maintains 100% codebook utilization throughout training, and surpasses current state-of-the-art VQNs on reconstruction quality on ImageNet, MNIST, and CIFAR-10.","pith_inferences":["The paper does not state how the row and column constraints in Eq. (8) are met when the feature count differs from the codebook size; in the reported setups the number of features is typically 16,384 while $n$ ranges down to 128, so the exact problem is infeasible. A useful stress test is to measure how the Sinkhorn column-sum error changes with $n/l$.","The experiments stop at reconstruction; if the same stability transfers to token-based generation, OptVQ could serve as a drop-in replacement in generative pipelines, which is the setting where index collapse is most damaging.","The manual choice of $\\epsilon$ in the paper suggests an interpretable direction: treat $\\epsilon$ as a temperature controlling assignment softness and anneal it during training, something the authors leave for future work."],"forward_implications":["VQNs can be trained from scratch on image reconstruction without the subtle initialization or model distillation that current methods use to avoid collapse.","Codebook utilization, which falls below 1% for large codebooks under nearest-neighbor VQ, stays at 100% across codebook sizes from 128 to 16,384 and latent dimensions 8 and 64.","Reconstruction quality improves with codebook size under OptVQ, whereas the conventional baseline's reconstruction error does not reliably fall as the codebook grows.","The quantization step remains cheap enough for practical training: roughly five Sinkhorn iterations are reported to converge, and a multi-head variant multiplies the effective codebook size to $n^B$.","When data and codebook distributions are already similar, OptVQ's assignment agrees with the nearest-neighbor assignment, so the method does not distort the final tokenization once training has stabilized."],"supporting_citations":[{"why":"Introduces the vector-quantized VAE and the nearest-neighbor quantizer whose local minima OptVQ targets.","marker":"[40]"},{"why":"Supplies the VQGAN architecture and composite loss used as the base for the ImageNet experiments.","marker":"[15]"},{"why":"Provides the Sinkhorn algorithm for entropy-regularized optimal transport that OptVQ runs.","marker":"[11]"},{"why":"Documents the straight-through estimator and initialization difficulties that motivate replacing nearest-neighbor search.","marker":"[21]"},{"why":"Identifies high-dimensional codebook sparsity and index collapse in vector-quantized image modeling.","marker":"[43]"},{"why":"Shows the distillation-based stabilization strategy that OptVQ aims to avoid.","marker":"[45]"},{"why":"Introduces the multi-head quantization mechanism OptVQ adopts to enlarge the effective codebook size.","marker":"[47]"},{"why":"Scales the codebook to 100,000 entries with 99% utilization, the closest baseline for the utilization comparison.","marker":"[48]"}],"fun_headline_variants":["OptVQ: Sinkhorn replaces nearest-neighbor, 100% codebook use","Optimal transport ends VQ's local-minima training trap","Sinkhorn assignment gives 100% codebook utilization in VQNs","From Voronoi to Sinkhorn: no more dead codes in VQ training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that five rounds of the Sinkhorn balancing procedure, run on a transport problem whose exact row and column balance is mathematically impossible whenever the number of features differs from the number of codes, still yield an assignment whose best code per feature keeps every codebook entry in use.","fun_headline_variants_meta":{"raw":{"variants":["OptVQ: Sinkhorn replaces nearest-neighbor, 100% codebook use","Optimal transport ends VQ's local-minima training trap","Sinkhorn assignment gives 100% codebook utilization in VQNs","From Voronoi to Sinkhorn: no more dead codes in VQ training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000882,"raw_usage":{"total_tokens":3771,"prompt_tokens":869,"completion_tokens":2902,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":2817}},"tokens_in":485,"tokens_out":2902,"duration_ms":18676,"temperature":1.0,"reasoning_tokens":2817,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:33:06.423378+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run OptVQ on a batch where the number of features $l$ is much larger than the codebook size $n$, record the column sums of the assignment matrix after five Sinkhorn iterations, and count how many codes receive zero argmax picks; if the column sums are far from uniform or any code is never selected, the claimed 100% utilization is not produced by the mechanism the paper describes.","supporting_citations":[{"cited_title":"Neural discrete representation learning","cited_arxiv_id":null,"evidence_quote":"Introduces the vector-quantized VAE and the nearest-neighbor quantizer whose local minima OptVQ targets."},{"cited_title":"Taming transformers for high-resolution image synthesis","cited_arxiv_id":null,"evidence_quote":"Supplies the VQGAN architecture and composite loss used as the base for the ImageNet experiments."},{"cited_title":"Sinkhorn distances: Lightspeed computation of optimal transport","cited_arxiv_id":null,"evidence_quote":"Provides the Sinkhorn algorithm for entropy-regularized optimal transport that OptVQ runs."},{"cited_title":"Straightening out the straight-through estimator: Over- coming optimization challenges in vector quantized net- works","cited_arxiv_id":null,"evidence_quote":"Documents the straight-through estimator and initialization difficulties that motivate replacing nearest-neighbor search."},{"cited_title":"Vector-quantized image modeling with improved vqgan","cited_arxiv_id":null,"evidence_quote":"Identifies high-dimensional codebook sparsity and index collapse in vector-quantized image modeling."},{"cited_title":"An image is worth 32 tokens for reconstruction and generation","cited_arxiv_id":null,"evidence_quote":"Shows the distillation-based stabilization strategy that OptVQ aims to avoid."},{"cited_title":"Movq: Modulating quantized vectors for high- fidelity image generation","cited_arxiv_id":null,"evidence_quote":"Introduces the multi-head quantization mechanism OptVQ adopts to enlarge the effective codebook size."}],"review_version":1}