{"id":"9f8574ec-3716-418c-9f5f-8a554c48c22d","arxiv_id":"2501.00877","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"FGAseg combines a pixel-text alignment transformer, a text-pixel alignment loss, and similarity-based pseudo-masks to achieve state-of-the-art open-vocabulary segmentation on multiple benchmarks.","lead":"FGAseg is a new model for open-vocabulary semantic segmentation that improves pixel-level alignment between images and text via cross-modal attention and text-pixel alignment losses. It also uses cosine and convolution-based similarity maps as pseudo-masks to add category boundary cues, reporting gains over prior methods on five common benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own results are inconsistent: §IV.C reports 37.7/63.1/96.9 mIoU for ViT-L/14 on A-150/PC-59/PAS-20 while Table I lists 37.9/63.4/97.1; the SOTA claim therefore rests on unverified numbers.","rationale":"FGAseg's central claim is empirical: it reports state-of-the-art mIoU on five benchmarks. For that claim to hold, the reported numbers must be accurate and reproducible. The paper does not provide code or detailed logs ('code is here' with no link), and the numbers that are given are not self-consistent. In Section IV.C, the ViT-L/14 results are quoted as 37.7 (A-150), 63.1 (PC-59), and 96.9 (PAS-20), while Table I shows 37.9, 63.4, and 97.1 for the same configuration—a discrepancy on three of five datasets. The ablation baselines are also inconsistent: Table II's no-module row (11.1 A-847) conflicts with Table III(a)'s 'w/o P2Tformer' frozen-CLIP baseline (8.5), and the kernel-size narrative (default 3 in §IV.D.1, '9 is optimal' in §IV.D.4) is contradicted by Table IV(a), where kernel 11 yields the highest A-847 and PC-459. These are not cosmetic typos; they affect the magnitude and even the direction of some ablation deltas. For example, adding Local Cost in Table II reduces A-847 from 12.1 to 12.0, while increasing PC-459 by 0.7. If the baseline numbers are off, the claimed contributions may be wrong. The reader's weakest assumption about pseudo-mask reliability is a reasonable methodological question, but it is secondary: even if pseudo-masks were noisy, the empirical SOTA claim could still hold if the overall architecture compensates. The immediate obstacle to accepting the claim is that the evidence presented for it is internally inconsistent. A conditional verdict is appropriate: the authors must release the code/checkpoints and correct the discrepancies before the SOTA claim can be taken as established.","tokens_in":17409,"tokens_out":7809,"duration_ms":60957,"concrete_test":"Obtain the authors' ViT-L/14 checkpoint and exact evaluation protocol (or have them release the full per-category results and logs). Recompute mIoU on A-150, PC-59, and PAS-20; then compare against both the §IV.C text numbers (37.7/63.1/96.9) and Table I (37.9/63.4/97.1). Also independently re-derive the Table II baseline row from the same frozen-CLIP setting as Table III(a). If any discrepancy exceeds rounding (0.1 mIoU), the reported SOTA claim is not supported by the available evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that FGAseg consistently outperforms SOTA is directly undermined by internal inconsistencies in the reported results. In §IV.C, the text states the ViT-L/14 model achieves 37.7 mIoU on A-150, 63.1 on PC-59, and 96.9 on PAS-20, while Table I lists 37.9, 63.4, and 97.1 for the same row. Additionally, the no-module baseline in Table II (11.1 on A-847) contradicts the 'w/o P2Tformer' frozen-CLIP baseline in Table III(a) (8.5), and §IV.D.1 sets the default kernel to 3 while §IV.D.4 claims kernel 9 is optimal (Table IV(a) shows 11 is best on several sets). These discrepancies mean the reported improvements, and thus the SOTA claim, rest on numbers that are not self-consistent. Without access to the code (the paper only says 'code is here' with no link) or a detailed evaluation log, a reader cannot determine which numbers are correct.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FGAseg, an open-vocabulary semantic segmentation approach that refines CLIP's image-level vision-text alignment through a Pixel-Level Alignment module, consisting of a Pixel-Text Alignment Transformer (P2Tformer) and a Text-Pixel Alignment Loss (T2Ploss). To preserve category boundaries, the method also introduces a Category Supplementation Propagation module that treats global cosine-similarity and local convolution-similarity maps as pseudo-masks during forward propagation. The model is trained end-to-end on COCO-Stuff and evaluated on A-847, PC-459, A-150, PC-59, and PAS-20, where the authors report consistent improvements over prior methods using ViT-B/16 and ViT-L/14 backbones, without external datasets or auxiliary backbones. The paper includes several ablation studies on the alignment module, kernel sizes, decoder variants, and training strategies.","tokens_in":1471,"tokens_out":1566,"duration_ms":49479,"significance":"If the reported numbers are correct, FGAseg provides a competitive and relatively simple recipe for open-vocabulary segmentation, achieving gains on multiple benchmarks while avoiding external datasets and auxiliary backbones. The architectural ideas - cross-modal pixel-text attention, a dedicated alignment loss, and using similarity maps as boundary-preserving pseudo-masks - are clear and reasonably motivated. The paper also includes extensive ablations and comparisons, which strengthens the empirical case. However, the overall credibility is undermined by several internal inconsistencies in the reported mIoU values and hyperparameter claims; these must be resolved before the stated state-of-the-art result can be assessed.","major_comments":[{"comment":"The main comparison text and Table I report different numbers for the same ViT-L/14 configuration: the text states mIoU of 37.7, 63.1, and 96.9 for A-150, PC-59, and PAS-20, while Table I lists 37.9, 63.4, and 97.1 for those same benchmarks. Because the paper's central claim is that FGAseg consistently outperforms existing methods, the reader cannot determine which of these values is the actual result. Please correct the discrepancy and specify which numbers are used in the claims.","section":"§IV.C and Table I"},{"comment":"The baseline configurations supposed to be the same are not consistent across the ablation tables. In Table II, the row with only the P2Tformer reports A-847 mIoU of 11.6 and the no-module baseline reports 11.1, while Table III(a) reports 8.5 for 'w/o P2Tformer' and 9.0 for 'w/ P2Tformer' on the same dataset and backbone. These differences of about 2.5-2.6 mIoU are too large to be ignored, and they directly affect the conclusion that the P2Tformer substantially improves alignment. Please reconcile the baselines and explain the cause of the discrepancy.","section":"Table II vs. Table III(a)"},{"comment":"The text states that a kernel size of 9 consistently achieves the highest mIoU, but Table IV(a) shows that kernel size 11 gives the best A-847 (12.1) and A-150 (32.0) scores and ties for the best PC-459 (19.4), while kernel 9 is not the maximum on any of these three sets. Moreover, §IV.D.1 sets the default kernel size to 3, yet the paper does not specify which kernel size is used in the final configuration for Table I. Please clarify the actual kernel size used in the main experiments and correct the description of the kernel-size dependency.","section":"§IV.D.4 and Table IV(a)"},{"comment":"The alignment loss Lalign is defined as an MSE between Oalign in R^{B x T x H x W} and M in R^{B x H x W}. As written, this subtraction is dimensionally incompatible unless M is broadcast or is actually a one-hot tensor with T channels. The text mentions 'a binary mask M' without specifying how it is expanded or whether the loss is computed per class. Please define the exact target representation and the reduction operation; otherwise the loss cannot be reproduced from the description.","section":"§III.B.2, Eq. (8)"}],"minor_comments":[{"comment":"The abstract and Section I state 'The code is here' without providing a URL or repository reference, and the Figure 7 caption says 'None of the datasets were seen by the model during training' despite the model being trained on COCO-Stuff and the figure displaying COCO-derived images. Please provide a working code link and clarify the relationship between COCO-Stuff training and the displayed COCO images.","section":"Abstract, §I, and Fig. 7 caption"},{"comment":"The text refers to 'the number of layers Nd in the P2Transformer', while elsewhere the module is called P2Tformer. Please use a single consistent notation throughout the paper.","section":"§IV.D.1"},{"comment":"In Table VI, the time for Top-N=16 on A-847 is reported as 256 ms, which is lower than the times for Top-N=4 (327 ms) and Top-N=8 (399 ms); this non-monotonic entry appears to be a typo and should be verified and corrected.","section":"Table VI"},{"comment":"The text says that incorporating T2Ploss results in 'slight improvements', but Table II shows that adding T2Ploss increases A-847 from 11.6 to 12.1 while PC-459 decreases from 18.4 to 18.3. The narrative should reflect this mixed outcome rather than uniformly implying improvement.","section":"§IV.D.2 and Table II"}],"recommendation":"major_revision","confidential_remarks":"The number of internal inconsistencies, especially between Table I and the main text and between Table II and Table III(a), is unusual and makes the state-of-the-art claim hard to verify without the code and a detailed training log. The authors should be asked to provide the missing code link, correct the numerical discrepancies, and confirm which hyperparameters were used in the main comparisons. If the discrepancies cannot be resolved, the central claim of consistent outperformance would need to be softened or removed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FGAseg is a reasonable recombination of known OVS components, but the paper's central SOTA claim is currently held up by numbers that don't agree with each other. That is the thing to know before spending time on it.\n\nWhat's actually new: not much conceptually. The P2Tformer cross-attention and the T2P loss are straightforward extensions of image-level CLIP alignment to pixel level. The cosine-similarity pseudo-masks are already in CAT-Seg and SED; the contribution is fusing them with a convolution-based local similarity and propagating that through a decoder. That is a credible engineering variant, and the paper describes it clearly. The visualization in Fig. 6 is a plus: it shows both successes and explicit failures of the pseudo-masks, which is more honest than most papers in this space. The training setup is clean: COCO-Stuff only, no extra backbones.\n\nThe problems are in the numbers. The ViT-L/14 results are reported as 37.7/63.1/96.9 in §IV.C but 37.9/63.4/97.1 in Table I. The no-module baseline is 11.1 in Table II but 8.5 in Table III(a). The default kernel size is set to 3 in §IV.D.1, while §IV.D.4 says 9 is optimal and Table IV(a) actually peaks at 11. These are not typos in one place; they are at least three independent contradictions across the main results and ablations. Together they undercut the claim that FGAseg consistently beats SOTA, because a reader can't tell which set of numbers is the real one. There is also no code link despite the text saying \"The code is here,\" and no error bars anywhere.\n\nNone of this makes the method itself incoherent. The architecture is sound, the losses are standard, and the reported gains are modest (1–3 mIoU). The soft spots are all in the reporting. But in this field, where benchmark numbers are the whole argument, those inconsistencies are load-bearing.\n\nWho this is for: someone working on open-vocabulary segmentation who wants a digestible summary of one more CLIP-alignment recipe. It could be a useful baseline, but only after the numbers are corrected and the code is released.\n\nMy recommendation: send it to peer review, but with a clear request for corrected tables, a code link, and error bars. If the numbers survive that, it's a citable incremental paper; if not, the SOTA claim should be withdrawn.","headline":"A plausible OVS recombination whose SOTA claim is currently undermined by internal numerical contradictions; fix the tables and it could be a citable incremental contribution.","tokens_in":18236,"tokens_out":2293,"would_cite":false,"duration_ms":20341,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FGAseg turns CLIP's coarse image-level alignment into pixel-level alignment and reports higher mIoU than prior open-vocabulary segmentation methods on five benchmarks.","keywords":["open-vocabulary semantic segmentation","vision-language models","CLIP","pixel-text alignment","pseudo-masks","category boundary","cross-modal attention","fine-grained alignment"],"falsifier":"Measure the mIoU drop on A-847 when the pseudo-mask branch is disabled at inference while keeping the aligned text embeddings; if the drop is small, the pseudo-mask mechanism is not the source of the reported gains. Alternatively, compare the pseudo-masks against ground-truth boundaries on a held-out set and check whether per-class agreement predicts per-class mIoU, since low agreement would contradict the paper's explanation.","tokens_in":17177,"feed_emoji":"🎯","tokens_out":8529,"duration_ms":68303,"temperature":0.7,"pith_summary":"This paper proposes FGAseg, a model for open-vocabulary semantic segmentation that turns CLIP's image-level vision-text alignment into fine-grained pixel-text alignment. The model adds a Pixel-Text Alignment Transformer, which lets text tokens attend to position-encoded image tokens, and a Text-Pixel Alignment Loss that supervises a local alignment map with ground-truth masks. It also treats cosine and convolution-based similarity matrices as pseudo-masks that carry category boundary information through the decoder. The authors report that FGAseg beats prior open-vocabulary segmentation methods on A-847, PC-459, A-150, PC-59, and PAS-20, reaching 16.3 mIoU on A-847 and 23.9 on PC-459 with a ViT-L/14 backbone while training only on COCO-Stuff.","feed_headline":"FGAseg tops open-vocabulary segmentation without extra data","feed_subtitle":"Training only on COCO-Stuff, it hits 16.3 mIoU on A-847 and 23.9 on PC-459 with ViT-L/14.","key_machinery":"The central object is a pair of similarity maps. The global map $S_g$ is the cosine similarity between aligned text embeddings and image features; the local map $S_l$ comes from sliding text-derived convolution kernels over the vision feature map, then normalizing the kernels with softmax. Fused, these maps act as a pseudo-mask that feeds class aggregation (a two-layer cross-attention using text guidance as query) and spatial aggregation (a Swin Transformer module). The second mechanism is the Pixel-Text Alignment Transformer (P2Tformer): a cross-attention layer in which text tokens query flattened, position-encoded image tokens, with a trainable scale $\\gamma$ blending the attended output into the original text embeddings. A Text-Pixel Alignment Loss computes mean squared error between an upsampled local alignment map and the ground-truth mask, steering the vision encoder toward pixel-level alignment without destroying the pretrained image-text alignment.","core_discovery":"FGAseg claims that the remaining gap between CLIP's pretraining and segmentation can be closed by a bidirectional alignment mechanism plus category-boundary supplementation, and that this is enough to outperform existing open-vocabulary segmentation models without external datasets or auxiliary backbones. The Pixel-Level Alignment module produces text embeddings that have looked at image content via cross-attention, and the Text-Pixel Alignment Loss forces local vision features to match ground-truth masks. The Category Supplementation Propagation module fuses a global cosine-similarity map with a local convolution-based similarity map into an optimizable pseudo-mask that guides class and spatial aggregation in the decoder. On five benchmarks, these components together yield higher mIoU than prior methods, with larger gains when scaling from ViT-B/16 to ViT-L/14.","pith_inferences":["Beyond the paper: the same similarity-map-as-pseudo-mask recipe could be lifted to other dense prediction tasks that need open-vocabulary grounding, such as referring-expression segmentation or open-vocabulary object detection, where boundary quality is also a bottleneck.","Beyond the paper: the paper does not measure pseudo-mask quality, so an immediate test is to compute the agreement of $S_g$ and $S_l$ with human-annotated boundaries on a held-out set and correlate that agreement with per-class mIoU.","Beyond the paper: the reliance on a fixed text prompt template ('a photo of a {class}') is likely a sensitivity point; evaluating the model with multiple prompt paraphrases could reveal how much of the reported gain depends on prompt wording."],"forward_implications":["Open-vocabulary segmentation can be improved on existing benchmarks without collecting new labeled data or adding auxiliary backbones, since the gains come from refining CLIP's own features.","The pseudo-mask propagation supplies boundary cues that benefit high-vocabulary benchmarks in particular: the method's largest relative gains appear on A-847 (16.3 mIoU) and PC-459 (23.9 mIoU).","The Top-K class selection at inference preserves most of the mIoU while cutting per-image time, so the accuracy gains do not have to come at the cost of real-time use.","The method scales with the vision backbone: switching from ViT-B/16 to ViT-L/14 raises mIoU on every reported dataset, suggesting the alignment components integrate cleanly with larger CLIP encoders."],"supporting_citations":[{"why":"Provides the frozen CLIP encoders whose image-level alignment FGAseg refines into pixel-level alignment.","marker":"[28]"},{"why":"CAT-Seg is the cost-aggregation method that first uses similarity matrices as pseudo-masks; FGAseg's global and local supplementation extends this idea.","marker":"[7]"},{"why":"SED contributes the encoder-decoder structure and the Top-K inference acceleration trick, and supplies the comparison numbers FGAseg benchmarks against.","marker":"[18]"},{"why":"OVSeg is a mask-adapted CLIP baseline that uses extra training data; FGAseg compares against it to show gains without external datasets.","marker":"[13]"},{"why":"SAN is a side-adapter baseline whose performance on the same benchmarks serves as a comparison point.","marker":"[8]"},{"why":"FC-CLIP is a single-stage frozen-CLIP baseline that demonstrates the difficulty of direct pixel-level prediction, against which FGAseg reports higher mIoU.","marker":"[26]"}],"fun_headline_variants":["FGAseg sharpens CLIP's coarse alignment to pixel level","Fine-grained alignment lifts open-vocab segmentation","FGAseg: pixel-text alignment beats open-vocab baselines","No extra data: FGAseg tops open-vocab segmentation","Closing the CLIP gap: FGAseg's fine-grained alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model's performance depends on the cosine and convolution similarity maps from CLIP being reliable guides to category boundaries for unseen classes; if those maps are noisy, the aggregation steps will amplify the noise rather than refine the mask.","fun_headline_variants_meta":{"raw":{"variants":["FGAseg sharpens CLIP's coarse alignment to pixel level","Fine-grained alignment lifts open-vocab segmentation","FGAseg: pixel-text alignment beats open-vocab baselines","No extra data: FGAseg tops open-vocab segmentation","Closing the CLIP gap: FGAseg's fine-grained alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000517,"raw_usage":{"total_tokens":2524,"prompt_tokens":977,"completion_tokens":1547,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":1459}},"tokens_in":593,"tokens_out":1547,"duration_ms":11499,"temperature":1.0,"reasoning_tokens":1459,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:42:16.196664+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the mIoU drop on A-847 when the pseudo-mask branch is disabled at inference while keeping the aligned text embeddings; if the drop is small, the pseudo-mask mechanism is not the source of the reported gains. Alternatively, compare the pseudo-masks against ground-truth boundaries on a held-out set and check whether per-class agreement predicts per-class mIoU, since low agreement would contradict the paper's explanation.","supporting_citations":[{"cited_title":"W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., & others","cited_arxiv_id":null,"evidence_quote":"Provides the frozen CLIP encoders whose image-level alignment FGAseg refines into pixel-level alignment."},{"cited_title":"H., & Kim, S","cited_arxiv_id":null,"evidence_quote":"CAT-Seg is the cost-aggregation method that first uses similarity matrices as pseudo-masks; FGAseg's global and local supplementation extends this idea."},{"cited_title":"S., & Pang, Y","cited_arxiv_id":null,"evidence_quote":"SED contributes the encoder-decoder structure and the Top-K inference acceleration trick, and supplies the comparison numbers FGAseg benchmarks against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"OVSeg is a mask-adapted CLIP baseline that uses extra training data; FGAseg compares against it to show gains without external datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SAN is a side-adapter baseline whose performance on the same benchmarks serves as a comparison point."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FC-CLIP is a single-stage frozen-CLIP baseline that demonstrates the difficulty of direct pixel-level prediction, against which FGAseg reports higher mIoU."}],"review_version":1}