REVIEW 4 major objections 5 minor 34 references
L2GNet: Optimal Local-to-Global Representation of Anatomical Structures for Generalized Medical Image Segmentation
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read L2GNet replaces self-attention with optimal-transport pooling and reports higher segmentation accuracy on two medical benchmarks.
desk verdict A plausible application of optimal-transport pooling to medical segmentation that is currently undermined by unverified baseline comparability and internal inconsistencies in the reported numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is the L2GMapper, a bottleneck that replaces the quadratic self-attention matrix with a linear-cost optimal-transport pooling step. It works by taking vector-quantized codes $z_{\rm dis}$, mapping them through a kernel embedding $\psi$ into an RKHS, computing the Sinkhorn transport plan $T(\psi(z_{\rm dis}), z_{\rm ref})$ between codes and a trainable reference $z_{\rm ref}$, and pooling the codes into $t$ bins with weights from the transport plan multiplied by a position kernel $S_{ij}=e^{-\frac{1}{\sigma^2_{\rm pos}}(i/n-j/t)^2}$. Multiple references $z^1_{\rm ref},\dots,z^q_{\rm ref}$ play the role of attention heads, and all Sinkhorn operations are differentiable, so the references are learned by backpropagation; the Nyström method is used as an approximation when code sets are large.
What would settle it
Re-run SynergyNet, TransUNet, SSNet, TranSSNet, and VQUNet in the exact L2GNet training configuration across multiple random seeds with identical data splits. If the mean Dice gaps on Synapse and ACDC shrink below run-to-run variance, or L2GNet no longer leads, the paper's central superiority claim is refuted. A complementary check: compute the transport plans and test whether removing the position kernel $S$ or replacing Sinkhorn with random assignment eliminates the reported gains.
Extended reading notes
Core claim
The central claim is that global context for segmentation can be obtained by optimal-transport alignment of discrete codes to a trainable reference rather than by self-attention. After the encoder produces a continuous latent and vector quantization converts it to discrete codes, the L2GMapper embeds those codes in a reproducing kernel Hilbert space, computes a Sinkhorn transport plan between the codes and a learnable reference, and pools codes into buckets weighted by both transport mass and a position-dependent similarity. The authors report that this procedure yields state-of-the-art results on Synapse and ACDC, with L2GNet(4-ref) reaching 82.23 mean DSC and 14.17 HD on Synapse and 91.44 DSC and 1.24 HD on ACDC, outperforming CLS, DLS, and CDLS baselines including SynergyNet, TransUNet, and VQUNet. They interpret the gains as evidence that the model learns inter-class and intra-class anatomical dependencies and reduces false negatives.
Load-bearing premise
The reported superiority assumes every baseline was measured under the same preprocessing, optimizer, epochs, data splits, and augmentation as L2GNet; the paper only states that it follows SynergyNet's configuration and does not show that the baseline numbers come from that same protocol.
Editorial extensions
If this is right
- Self-attention bottlenecks in segmentation networks can be replaced by a linear-complexity optimal-transport pooling module without sacrificing accuracy on multi-organ and cardiac tasks.
- The reported margins over SynergyNet, TransUNet, and VQUNet imply that modeling dependencies between pertinent regions, rather than all pairwise patch interactions, is sufficient for strong segmentation performance.
- Codebook size and the number of references are tunable and affect results: L2GNet peaks at $K=512$ and $q=4$ in the reported experiments, and performance degrades at smaller codebooks and at $q=2$.
- Because the module is a drop-in bottleneck, the same design could be attached to other encoders and decoders, potentially improving generalization on additional anatomy segmentation tasks.
Reading between the lines
- Editorial inference: the comparison would be strongest if all baselines were re-trained under L2GNet's exact protocol with multiple seeds, since the paper's tables do not report variance and only state that SynergyNet's configuration was followed.
- Editorial inference: the position-aware transport plan could be visualized as attention maps to test the claim that it focuses on pertinent regions and avoids redundant pairs; the paper does not provide such a quantitative analysis.
- Editorial inference: the same optimal-transport bottleneck may extend to 3D volumes and semi-supervised settings, where the authors' claimed linear complexity and annotation efficiency would matter most.
- Editorial inference: the method's benefit may depend on the entropy regularization $\varepsilon$ in Sinkhorn and the position-kernel width $\sigma_{\rm pos}$; ablating those would clarify whether the gain comes from optimal transport or from the added positional prior.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes L2GNet, a medical image segmentation architecture that combines vector-quantized discrete representations with a trainable optimal-transport (OT) embedding to replace dot-product self-attention. The authors argue that CLS, DLS, and CDLS methods either miss long-range dependencies or capture redundant ones, and that L2GNet's OT-based pooling onto learned references yields discriminative global representations at linear cost. The method is evaluated on the Synapse multi-organ and ACDC cardiac datasets, reporting improvements over several baselines including the authors' own SynergyNet. The central empirical claim is that L2GNet outperforms state-of-the-art CLS, DLS, and CDLS methods, but the experimental validation is incomplete and internally inconsistent.
Significance. If the central claim were rigorously established, L2GNet would be a useful contribution: it offers a concrete alternative to quadratic self-attention bottlenecks, is applicable to 2D medical image segmentation, and its integration of VQ with an OT embedding is a plausible way to encourage structured, anatomy-aware aggregation. The paper explicitly builds on the trainable OT embedding of Mialon et al. [22] and Sinkhorn's algorithm [21], and the novelty lies in adapting these tools for discrete-code aggregation in segmentation rather than inventing a new OT formalism. The complexity argument (reducing the attention matrix from O(n^2) to O(n)) is reasonable, and the use of multiple references resembles multi-head attention. However, the significance is currently limited by the lack of controlled experiments, missing error bars, and several presentation defects. No code or trained models are released, which further hampers verification of the empirical claims.
major comments (4)
- [Section 4, Dataset and Experiment Settings] The experimental section never specifies the preprocessing and training configuration claimed to follow SynergyNet [16] (e.g., input resolution, optimizer, learning rate, batch size, number of epochs, loss terms, augmentation, and official train/validation splits), nor does it state whether the CLS/DLS/CDLS baselines in Tables 1 and 2 were retrained under L2GNet's protocol or their numbers were taken from the original publications. Because SynergyNet, SSNet, and TranSSNet are prior work by the same authors, a controlled re-run of all baselines in the same setting is feasible and is required to support the claimed superiority; otherwise the margins in Tables 1 and 2 may reflect protocol differences rather than architectural benefit.
- [Tables 1 and 2] No error bars, confidence intervals, or significance tests are reported for any metric. The ACDC DSC margin between L2GNet(4-ref) and TranSSNet is only 0.12 points (91.44 vs 91.32), and between L2GNet(4-ref) and SynergyNet is 1.66 points; with the run-to-run variance typical of medical image segmentation, these differences cannot be judged without multiple seeds. The HD metric is particularly sensitive to outliers and should be reported with variance.
- [Tables 1, 3, 4, and 5] The baseline numbers are internally inconsistent: SynergyNet's Synapse HD is 23.59 in Table 1 but 23.29 in Tables 4 and 5, and its Synapse DSC is 79.65 in Table 1 but 79.61 in Table 3. This suggests that baseline scores were not produced by a single controlled evaluation protocol, and it directly undermines the credibility of the comparisons in the main tables. The authors should rerun all baselines in exactly the L2GNet setting and report a single consistent set of numbers.
- [Section 3, Eq. (3)] Equation (3), which defines the proposed embedding phi_{z_ref}(z_dis), is not readable as typeset: it mixes the indices 1i and tit, contains an unexplained '×S,...,' and the expression is not a well-formed vector. In addition, the preceding sentence defines the cost as M = -psi(z_dis), but M should be a pairwise cost matrix between codes and reference elements. Because this equation is the core of the proposed L2GMapper, it must be rewritten precisely and consistently.
minor comments (5)
- [Abstract and Keywords] The listed keywords ('Cirrhotic liver segmentation, Abdominal MRI dataset, liver segmentation, ...') do not match the paper's content, which concerns multi-organ and cardiac segmentation via optimal transport; these keywords should be corrected.
- [Tables 3 and 6] Tables 3 and 6 are identical (codebook embedding size analysis) and should be merged or one should be removed.
- [Abstract] The phrase 'an novel approach' contains a grammatical error; it should be 'a novel approach.'
- [Section 2, Eq. (1)] The notation is inconsistent: the text introduces pairwise costs C, but Eq. (1) uses M_{ij}; the variable should be matched to avoid ambiguity.
- [References] Reference [1] (Kawaguchi et al.) appears not to be cited in the body of the paper; either cite it or remove it.
Circularity Check
No significant circularity: the core method is an external OT embedding and the claims are benchmark evaluations.
full rationale
The derivation chain of L2GNet is not circular. The L2GMapper module relies on a trainable optimal transport embedding, and the paper explicitly attributes that embedding and the Nyström approximation to external references ([21], [22], [23], [25]) rather than to the authors' own prior work. The main claims are empirical results on the Synapse and ACDC benchmarks. No fitted constant is relabeled as a prediction, and no output is defined in terms of the quantity it is supposed to predict. The paper does use the authors' own SynergyNet/SSNet/TranSSNet as baselines and cites SynergyNet [16] for the training configuration, but this is not a load-bearing mathematical self-citation: the comparison numbers are benchmark scores, and the configuration reference affects experimental protocol, not the logical derivation of the method. Concerns that baseline numbers may have been produced under different training protocols, and the internal inconsistencies in the reported SynergyNet values across Tables 1, 3, 4, and 5, are experimental-comparability and reporting-quality issues, not circularity, because they do not make L2GNet's result equivalent to its inputs by construction. No circular step can be exhibited by quoting a specific equation whose output is identical to its input by definition. The paper is self-contained against external benchmarks, so the honest finding is no significant circularity.
Assumptions & free parameters
free parameters (5)
- Codebook size K =
512
- Latent dimension dim =
1024
- Number of references q =
4
- Sinkhorn iterations =
10
- Positional scaling sigma_pos =
not reported
assumptions (4)
- standard math Sinkhorn's algorithm converges to a unique solution of the entropic OT problem (Eq. 1).
- domain assumption The linear-kernel Nystrom approximation of the RKHS embedding is faithful.
- domain assumption Vector quantization preserves enough anatomical information for downstream optimal-transport pooling.
- domain assumption The positional weighting matrix S encodes meaningful spatial relationships and improves alignment.
Cite this review
Pith. "Pith review of L2GNet: Optimal Local-to-Global Representation of Anatomical Structures for Generalized Medical Image Segmentation." pith.science (2026). https://pith.science/paper/PSKGBXE2
@misc{pith2026250205229,
author = {Pith},
title = {Pith review of: L2GNet: Optimal Local-to-Global Representation of Anatomical Structures for Generalized Medical Image Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/PSKGBXE2}},
note = {Machine review of arXiv:2502.05229}
}
read the original abstract
Continuous Latent Space (CLS) and Discrete Latent Space (DLS) models, like AttnUNet and VQUNet, have excelled in medical image segmentation. In contrast, Synergistic Continuous and Discrete Latent Space (CDLS) models show promise in handling fine and coarse-grained information. However, they struggle with modeling long-range dependencies. CLS or CDLS-based models, such as TransUNet or SynergyNet are adept at capturing long-range dependencies. Since they rely heavily on feature pooling or aggregation using self-attention, they may capture dependencies among redundant regions. This hinders comprehension of anatomical structure content, poses challenges in modeling intra-class and inter-class dependencies, increases false negatives and compromises generalization. Addressing these issues, we propose L2GNet, which learns global dependencies by relating discrete codes obtained from DLS using optimal transport and aligning codes on a trainable reference. L2GNet achieves discriminative on-the-fly representation learning without an additional weight matrix in self-attention models, making it computationally efficient for medical applications. Extensive experiments on multi-organ segmentation and cardiac datasets demonstrate L2GNet's superiority over state-of-the-art methods, including the CDLS method SynergyNet, offering an novel approach to enhance deep learning models' performance in medical image analysis.
Figures
Reference graph
Works this paper leans on
-
[22]
A Trainable Optimal Transport Embedding for Feature Aggregation and its Relationship to Attention
G. Mialon et al., “A trainable optimal transport em- bedding for feature aggregation and its relationship to attention,” arXiv preprint arXiv:2006.12065, 2020
work page Pith review arXiv 2006
-
[21]
Sinkhorn distances: Lightspeed com- putation of optimal transport,
M. Cuturi, “Sinkhorn distances: Lightspeed com- putation of optimal transport,” Advances in Neural Information Processing Systems, vol. 26, 2013
work page 2013
-
[16]
V . Gorade, S. Mittal, D. Jha, and U. Bagci, “Synergy- Net: Bridging the gap between discrete and continuous representations for precise medical image segmenta- tion,” in Proc. WACV, pp. 7768–7777, 2024. 7
work page 2024
-
[1]
Generalization in deep learning,
K. Kawaguchi, L. P. Kaelbling, and Y . Bengio, “Generalization in deep learning,” arXiv preprint 6 Table 6: Codebook embedding size analysis. Kdim Synapse ACDC SynergyNet L2GNet SynergyNet L2GNet DSC HD DSC HD DSC HD DSC HD 1024 77.61 29.53 80.27 24.00 88.64 2.12 91.41 1.36 512 79.61 23.89 82.23 14.17 88.89 1.86 91.44 1.24 256 79.21 30.07 81.35 22.92 89.1...
-
[2]
Survey on deep learning for radiotherapy,
P. Meyer, V . Noblet, C. Mazzara, and A. Lallement, “Survey on deep learning for radiotherapy,”Computers in Biology and Medicine , vol. 98, pp. 126–146, 2018
work page 2018
-
[3]
Z. Yan, X. Han, C. Wang, Y . Qiu, Z. Xiong, and S. Cui, “Learning mutually local-global U-nets for high- resolution retinal lesion segmentation in fundus im- ages,” in Proc. IEEE ISBI, pp. 597–600, 2019
work page 2019
-
[4]
W. Chen, Z. Jiang, Z. Wang, K. Cui, and X. Qian, “Col- laborative global-local networks for memory-efficient segmentation of ultra-high resolution images,” inProc. IEEE/CVF CVPR, pp. 8924–8933, 2019
work page 2019
-
[5]
K. Chaitanya, E. Erdil, N. Karani, and E. Konukoglu, “Contrastive learning of global and local features for medical image segmentation with limited annotations,” Advances in Neural Information Processing Systems , vol. 33, pp. 12546–12558, 2020
work page 2020
Show all 34 references
-
[6]
Attention is all you need,
A. Vaswani et al., “Attention is all you need,”Advances in Neural Information Processing Systems , vol. 30, 2017
2017
-
[7]
An image is worth 16x16 words: Transformers for image recognition at scale,
A. Dosovitskiy et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020
2010 arXiv
-
[8]
Swin transformer: Hierarchical vi- sion transformer using shifted windows,
Z. Liu et al., “Swin transformer: Hierarchical vi- sion transformer using shifted windows,” in Proc. IEEE/CVF ICCV, pp. 10012–10022, 2021
2021
-
[9]
UNet ++: A nested U-Net architecture for medical image segmentation,
Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “UNet ++: A nested U-Net architecture for medical image segmentation,” in Proc. Int. Workshop Deep Learning Med. Image Analysis , pp. 3–11, 2018
2018
-
[10]
Attention U-Net: Learning where to look for the pancreas,
O. Oktay et al., “Attention U-Net: Learning where to look for the pancreas,” arXiv preprint arXiv:1804.03999, 2018
2018 arXiv
-
[11]
Swin-UNet: UNet-like pure trans- former for medical image segmentation,
H. Cao et al., “Swin-UNet: UNet-like pure trans- former for medical image segmentation,” in Proc. ECCV, pp. 205–218, 2022
2022
-
[12]
TransUNet: Transformers make strong encoders for medical image segmentation,
J. Chen et al., “TransUNet: Transformers make strong encoders for medical image segmentation,” arXiv preprint arXiv:2102.04306, 2021
2021 arXiv
-
[13]
Vector quantisation for robust segmentation,
A. Santhirasekaram et al., “Vector quantisation for robust segmentation,” in Proc. MICCAI, pp. 663–672, 2022
2022
-
[14]
Neural discrete representation learning,
A. Van Den Oord and O. Vinyals, “Neural discrete representation learning,” Advances in Neural Informa- tion Processing Systems, vol. 30, 2017
2017
-
[15]
HiFormer: Hierarchical multi- scale representations using transformers for medical image segmentation,
M. Heidari et al., “HiFormer: Hierarchical multi- scale representations using transformers for medical image segmentation,” in Proc. WACV, pp. 6202–6212, 2023
2023
-
[17]
Harmonized spatial and spectral learning for robust and generalized medical image segmentation,
V . Gorade, S. Mittal, D. Jha, R. Singhal, and U. Bagci, “Harmonized spatial and spectral learning for robust and generalized medical image segmentation,” arXiv preprint arXiv:2401.10373, 2024
2024 arXiv
-
[18]
Supervised contrastive learning,
P. Khosla et al., “Supervised contrastive learning,” Advances in Neural Information Processing Systems , vol. 33, pp. 18661–18673, 2020
2020
-
[19]
Domain adaptive relational reasoning for 3D multi-organ segmentation,
S. Fu et al., “Domain adaptive relational reasoning for 3D multi-organ segmentation,” in Proc. MICCAI, pp. 656–666, 2020
2020
-
[20]
U-Net: Convolutional networks for biomedical image segmen- tation,
O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmen- tation,” in Proc. MICCAI, pp. 234–241, 2015
2015
-
[23]
Using the Nystr ¨om method to speed up kernel machines,
C. Williams and M. Seeger, “Using the Nystr ¨om method to speed up kernel machines,” Advances in Neural Information Processing Systems, vol. 13, 2000
2000
-
[24]
Computational opti- mal transport,
G. Peyr ´e, M. Cuturi, et al., “Computational opti- mal transport,” Center for Research in Economics and Statistics Working Papers, no. 2017-86, 2017
2017
-
[25]
End-to-end kernel learning with super- vised convolutional kernel networks,
J. Mairal, “End-to-end kernel learning with super- vised convolutional kernel networks,” Advances in Neural Information Processing Systems, vol. 29, 2016
2016
-
[26]
Biological se- quence modeling with convolutional kernel networks,
D. Chen, L. Jacob, and J. Mairal, “Biological se- quence modeling with convolutional kernel networks,” Bioinformatics, vol. 35, no. 18, pp. 3294–3302, 2019
2019
-
[27]
[Online]
Multi-Atlas Abdomen Labeling Challenge: Synapse Multi-Organ Segmentation Dataset , Synapse Consortium, 2015. [Online]. Avail- able: https://www.synapse.org/#!Synapse: syn3193805/wiki/217789
2015
-
[28]
ACDC (Automated Cardiac Diagnosis Challenge) ,
-
[29]
Mean teachers are bet- ter role models: Weight-averaged consistency targets improve semi-supervised deep learning results,
A. Tarvainen and H. Valpola, “Mean teachers are bet- ter role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” Ad- vances in Neural Information Processing Systems , vol. 30, 2017
2017
-
[30]
Advent: Adversarial entropy mini- mization for domain adaptation in semantic segmen- tation,
T.-H. Vu et al., “Advent: Adversarial entropy mini- mization for domain adaptation in semantic segmen- tation,” in Proc. IEEE /CVF CVPR , pp. 2517–2526, 2019
2019
-
[31]
Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmenta- tion,
L. Yu et al., “Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmenta- tion,” in Proc. MICCAI, pp. 605–613, 2019
2019
-
[32]
Interpolation consistency training for semi-supervised learning,
V . Verma et al., “Interpolation consistency training for semi-supervised learning,” Neural Networks, vol. 145, pp. 90–106, 2022
2022
-
[33]
Semi-supervised semantic seg- mentation with cross pseudo supervision,
X. Chen et al., “Semi-supervised semantic seg- mentation with cross pseudo supervision,” in Proc. IEEE/CVF CVPR, pp. 2613–2622, 2021. 8
2021
-
[2017]
Available: https://www.creatis
[Online]. Available: https://www.creatis. insa-lyon.fr/Challenge/acdc
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.