{"id":"4c39fd8a-111d-4baf-8757-aa3a6daa73f5","arxiv_id":"2411.11357","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The authors introduce ZSOLNet, a CLIP-based text-guided zero-shot object localization framework with a text self-similarity matching module, evaluated on FSC-147, CARPK, and ShanghaiTech.","lead":"This paper presents a method to locate objects in images using text descriptions instead of labeled examples, combining CLIP with a text feature enhancement module. It reports competitive results on three public datasets, but the gains are modest and the approach extends existing zero-shot counting pipelines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TSSM's claimed benefit rests on a single unseeded ablation with internally inconsistent counting results; the module's contribution is not statistically established.","rationale":"The reader's weakest assumption identifies the TSSM heuristic as empirically unsupported, and my review agrees: this is the most load-bearing concern because TSSM is the paper's main novel contribution. The ablation is the only direct evidence, and it lacks the statistical controls needed to distinguish a real effect from noise or threshold artifacts. The internal inconsistency in the table (localization improves while MAE worsens) further weakens the claim that TSSM enhances text representations generally. No formal verification, code release, or independent replication is provided, so the concern cannot be dismissed. The zero-shot framing also has wording issues—training on FSC-147 with ground-truth density maps is standard for class-agnostic zero-shot counting, and the paper's 'without any labeled information' phrasing in the conclusion is overbroad—but that is less load-bearing than the unestablished TSSM benefit. Keeping the reader's CONDITIONAL verdict is appropriate: the concern is real, but it is addressable with additional experiments and clarifications rather than requiring rejection.","tokens_in":15870,"tokens_out":6007,"duration_ms":66436,"concrete_test":"Rerun the TSSM on/off ablation on FSC-147 with at least 5 random seeds, keeping all other hyperparameters fixed, and report mean±std for F1/AP/AR/MAE/MSE at both thresholds, plus a paired bootstrap or Wilcoxon signed-rank test on per-image F1 at σs=5. If the confidence intervals overlap or the paired test gives p>0.05, the TSSM contribution is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novel component is the TSSM module, but its only supporting evidence is Table V, a one-run ablation on FSC-147 with no seeds, error bars, or significance tests. The table is internally mixed: with TSSM, localization F1/AP/AR improve at both thresholds, but MAE degrades on validation (28.46 vs 24.33) and test (20.78 vs 18.21), while test MSE barely changes (100.14 vs 105.43). This pattern suggests TSSM may sharpen density peaks rather than improve true localization, and the F1 gains could interact with the post-processing thresholds. Additionally, Eqs. (4)-(5) are underspecified: the dimensions of εt and εo are not defined, 'n=3' is described as batch size but used as a summation index, and the claimed convolution/title-embedding extraction is not written out. The module is therefore not reproducible from the text, and the load-bearing claim that TSSM improves localization accuracy is not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Zero-Shot Object Localization (ZSOL), a framework that uses CLIP-based visual and textual encoders together with a Text Self-Similarity Matching (TSSM) module to localize objects from prompt words without class-specific labeled examples. The model is trained on FSC-147 with frozen encoders, a contrastive loss, and an MSE loss on predicted density maps, then post-processes density peaks with two thresholds. The authors report localization and counting results on FSC-147, ShanghaiTech A/B, and CARPK, comparing against fully supervised and few-shot methods, and conclude that ZSOL achieves performance comparable to few-shot approaches while requiring no annotated target-category data.","tokens_in":16093,"tokens_out":2701,"duration_ms":29882,"significance":"If the claims are substantiated, the paper would provide a useful first benchmark for text-guided zero-shot object localization, which is a plausible extension of recent zero-shot counting work. The use of frozen CLIP encoders, a text self-similarity module, and density-map post-processing is a reasonable design that could be a practical baseline. The paper includes comparisons across several standard datasets and reports both localization and counting metrics. However, the central novel component, TSSM, is not specified precisely enough to be reproduced, and its only supporting ablation is a single-run experiment with mixed and statistically unvalidated results. The per-dataset threshold tuning also weakens the zero-shot claim. These issues currently prevent the paper from being accepted as a reliable technical contribution.","major_comments":[{"comment":"The TSSM module is not specified precisely enough to be implemented. The dimensions of εt and εo are never defined, and the statement that \"n = 3 represents three pairs of samples fed into each batch\" is inconsistent with n being used as the summation index in Eq. (4), where n should be the embedding dimension. The convolution mentioned in the text (\"The title embedding obtained by the convolution is again computed by cosine similarity\") is never written out: the kernel, stride, padding, and input/output shapes are all absent. Because TSSM is the paper's main novelty, these missing definitions are load-bearing.","section":"§III-B1, Eqs. (4)-(5)"},{"comment":"The evidence that TSSM improves localization is not convincing. The ablation shows F1/AP/AR gains with TSSM, but MAE is consistently worse with the module (28.46 vs. 24.33 on validation, 20.78 vs. 18.21 on test), while test MSE is essentially unchanged (100.14 vs. 105.43). This pattern suggests TSSM may sharpen density peaks rather than improve density-map accuracy. The ablation is performed on one dataset, without multiple seeds, error bars, or significance tests. The claim that \"TSSM has a positive effect on improving zero-shot object localization performance\" is therefore not statistically established.","section":"Table V (ablation study)"},{"comment":"The training objectives and decoder architecture are not described precisely. The \"contrastive loss\" and \"MSE loss\" are only described in words; no equations, weighting coefficients, or exact definitions are given. The \"joint embedding decoder\" that maps multimodal features to a density map is not architecturally specified. The two-stage training procedure (20 contrastive epochs followed by 200 MSE epochs) is also underspecified with regard to batch size, image crop size, and learning-rate schedule. As a result, the experimental section is not reproducible from the manuscript alone.","section":"§III-B2 and §III-B3"},{"comment":"The peak thresholds α and β are selected per dataset and are not derived from a validation procedure. The text states that α = 5/255 is used for dense datasets and α = 10/255 for sparse datasets, and β = 0.06 globally. Because these thresholds are tuned on the test datasets and no sensitivity analysis is provided, it is unclear how much of the reported performance depends on this manual calibration. This weakens the zero-shot generalization claim, since per-dataset threshold tuning is a form of test-set adaptation.","section":"§III-B3 (post-processing thresholds)"},{"comment":"The comparison with prior methods is not always apples-to-apples. For baselines without a double-threshold setting, the paper uses their reported SOTA results, while for others it uses thresholds σs and σl that may differ from the original papers; the exact provenance of each baseline number is not itemized. Additionally, the ZSOL model is trained on a cropped-image protocol and tested with a sliding window, which the paper itself notes can inflate test performance relative to validation. The tables show many missing entries for baselines, making it difficult to assess whether the reported gains are consistent across settings.","section":"§IV-D and Tables I-IV"}],"minor_comments":[{"comment":"The notation Wit = f(Ie, Te) is introduced without explaining the projection layer f(·) or the dimensionality of Wit; please define all symbols in this equation.","section":"Eq. (2)"},{"comment":"The terms \"title\", \"prompt word\", and \"sentence\" are used interchangeably; clarify the relationship between the user-provided prompt, the expanded sentence, and the extracted title embedding.","section":"§III-B1 and Fig. 4"},{"comment":"The column layout of Table V is corrupted in the displayed text, with \"MAE MSE√\" on one line and the numeric values not clearly aligned; please reformat the table so that validation/test splits and metric columns are unambiguous.","section":"Table V"},{"comment":"There is a typo in the sentence \"the ZSOL model exhibits excellent localization performance in scenarios where the object object localization is highly distinguished from the background\"; remove the duplicated \"object\".","section":"§IV-D"},{"comment":"The conclusion calls the framework \"a high-performance benchmark\" but no code, model weights, or evaluation protocol are released; please state whether these will be made available.","section":"§V"}],"recommendation":"major_revision","confidential_remarks":"The paper's novelty claim of being the first text-guided zero-shot object localization method should be checked more carefully against prior text-guided counting works (e.g., CLIP-Count, CountGD) that also produce point-level localizations. The authors cite these works but position them as counting only; a clearer positioning analysis would help the editor assess the contribution. The technical reproducibility issues and the weak statistical evidence for TSSM are serious enough to require major revision before the paper can be considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is best read as an engineering extension of the CLIP-based zero-shot counting line (CLIP-Count, VLCounter) to the localization task. The genuinely new piece is the TSSM module, and the evidence for it is thinner than the authors claim. On the plus side, the two-stage training (contrastive loss for patch-level alignment, then MSE for density regression) is reasonable, and the results on ShanghaiTechA and CARPK are competitive with few-shot methods, which is worth something. The paper also tests on three datasets with different density profiles, which helps.\n\nThe soft spots are real. The \"for the first time\" novelty claim is overblown, given the cited zero-shot counting methods already produce density maps that can be localized. TSSM is an adaptation of the SSP self-support idea, not a new paradigm. More importantly, the module's benefit is not established: Table V is one run, no seeds, no error bars, and the MAE gets worse when TSSM is added (28.46 vs 24.33 on validation, 20.78 vs 18.21 on test). That pattern suggests TSSM sharpens peaks rather than improving true localization, and the F1 gains might interact with the post-processing thresholds. The math is also under-specified: the convolution in Eq. (5) is never defined, \"n=3\" is called a batch size but used as a summation index, and the title embedding extraction is a black box. I could not reproduce the module from the text.\n\nThe per-dataset threshold tuning is defensible but should be reported as part of the method, not as a surprise. The authors do flag the validation/test discrepancy from cropping versus sliding window, which is honest.\n\nThe paper deserves a serious referee because the task is useful and the empirical coverage is decent, but the referee should push for a rewritten methodology section, multiple seeds, and a head-to-head with CLIP-Count under identical post-processing. I would not cite it in its current form, but I'd keep an eye on a revised version.","headline":"An engineering extension of zero-shot counting with an under-supported TSSM module; the paper's own ablation shows counting metrics degrading, so the central claim needs much better evidence.","tokens_in":16602,"tokens_out":2603,"would_cite":false,"duration_ms":24756,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that text-guided zero-shot object localization works by aligning prompt-word-enhanced CLIP text embeddings with image patches, producing density maps whose peaks are object centers without labeled examples.","keywords":["zero-shot object localization","text-guided localization","CLIP","text self-similarity matching","density map estimation","few-shot object localization","cross-modal learning","FSC-147"],"falsifier":"Run the same ZSOL pipeline with and without TSSM on multiple seeds and datasets, then compare F1 distributions at both thresholds; if the gap is not consistently positive or overlaps zero, the module's claimed contribution fails. A simpler probe: replace Eq. (5) with concatenation or simple averaging of the two embeddings; if localization accuracy does not drop, the similarity-weighting mechanism is not doing the claimed work.","tokens_in":15706,"feed_emoji":"📍","tokens_out":5392,"duration_ms":48965,"temperature":0.7,"pith_summary":"The paper sets out to prove that object localization can be done with zero labeled examples, using only a text prompt describing the target object. It builds a framework around a frozen CLIP model, which already maps images and text into a shared space, and adds a text self-similarity matching (TSSM) module that weights text features toward the prompt word. The authors report that the resulting ZSOLNet locates objects on FSC-147, CARPK, and ShanghaiTech with accuracy comparable to recent few-shot localization methods, and sometimes above fully supervised ones in dense scenes. If the result holds, it would let users localize arbitrary categories by typing a phrase, removing the annotation bottleneck.","feed_headline":"Text prompts locate objects with zero labeled samples","feed_subtitle":"A CLIP-based model reads a phrase and outputs object locations, rivaling few-shot methods on standard benchmarks.","key_machinery":"The central mechanism is the Text Self-Similarity Matching (TSSM) module, which takes the sentence-level text embedding $\\varepsilon_t$ and the prompt-word embedding $\\varepsilon_o$, computes their cosine similarity $W$, and forms the self-supporting embedding $\\varepsilon_{ts} = W \\cdot \\varepsilon_t + \\varepsilon_o$. This embedding then replaces the original text feature in the CLIP alignment. The other machinery is a two-stage training loop: first a contrastive loss aligns image patches near density peaks with the text embedding, then an MSE loss refines the predicted density map; post-processing thresholds $\\alpha$ and $\\beta$ extract final object centers.","core_discovery":"This paper claims the first zero-shot object localization framework driven purely by prompt words: given an image and a text phrase, the model outputs a density map, and local maxima mark object centers. The central discovery is that re-weighting the CLIP text embedding by its cosine similarity to the prompt-word embedding (the TSSM module) substantially improves localization precision, and that a frozen CLIP backbone can be adapted to dense localization through contrastive patch-level pretraining followed by MSE training. Reported results on FSC-147, CARPK, and ShanghaiTech place the zero-shot model within the range of few-shot and fully supervised methods, and on ShanghaiTechA it exceeds several supervised baselines on F1 and AP.","pith_inferences":["An untested extension is to read the density map's integral as a count, turning the same model into a zero-shot counter; this would unify two tasks the paper treats separately.","The 'first zero-shot localization' claim is narrower than it sounds: earlier text-guided zero-shot counting methods already used CLIP-style prompts, so the new contribution is applying the idea to center-point localization, not inventing cross-modal prompting.","A stress test the paper does not report is varying the prompt wording (synonyms, plurals, or captions instead of noun phrases); TSSM's cosine weighting may be sensitive to phrasing, which would matter in practice.","The method inherits CLIP's training distribution, so its zero-shot scope is bounded by the concepts CLIP has seen; testing on genuinely novel object categories would reveal the practical boundary."],"forward_implications":["Users could localize any object category by typing a phrase, with no per-category training data or annotation effort.","The zero-shot model is competitive in dense scenes, matching or exceeding few-shot baselines on ShanghaiTechA and CARPK while staying close on FSC-147.","Removing the TSSM module drops F1 substantially on FSC-147, so the text re-weighting step is a necessary part of the reported accuracy.","Because the CLIP encoders stay frozen, the same trained decoder transfers to new datasets and categories without retraining."],"supporting_citations":[{"why":"Supplies the frozen CLIP visual and text encoders that map prompt words and images into a shared embedding space.","marker":"[26]"},{"why":"Inspires the TSSM module: the self-support matching idea from few-shot semantic segmentation is adapted to re-weight text features.","marker":"[27]"},{"why":"Defines the few-shot object localization task and provides the FSOL baseline whose results ZSOL compares against on FSC-147 and ShanghaiTech.","marker":"[39]"},{"why":"Provides the focal inverse distance transform maps and the two-threshold post-processing used to turn density maps into object centers.","marker":"[43]"},{"why":"Shows a text-guided zero-shot counting method that the paper positions as related work, establishing the multimodal prompt paradigm ZSOL extends.","marker":"[21]"}],"fun_headline_variants":["Prompt words alone localize objects with zero labeled samples","Zero-shot object localization using CLIP text self-similarity","Text-only model pinpoints objects without any annotations","CLIP prompt matching achieves zero-shot object localization","Zero-shot localization from a single text phrase"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The TSSM module's core assumption is that adding the prompt-word embedding weighted by cosine similarity to the sentence embedding makes the text representation sharper for localization; this is an empirical heuristic, tested on one dataset without statistical significance checks.","fun_headline_variants_meta":{"raw":{"variants":["Prompt words alone localize objects with zero labeled samples","Zero-shot object localization using CLIP text self-similarity","Text-only model pinpoints objects without any annotations","CLIP prompt matching achieves zero-shot object localization","Zero-shot localization from a single text phrase"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1374,"prompt_tokens":837,"completion_tokens":537,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":463}},"tokens_in":453,"tokens_out":537,"duration_ms":5702,"temperature":1.0,"reasoning_tokens":463,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:36:29.021403+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same ZSOL pipeline with and without TSSM on multiple seeds and datasets, then compare F1 distributions at both thresholds; if the gap is not consistently positive or overlaps zero, the module's claimed contribution fails. A simpler probe: replace Eq. (5) with concatenation or simple averaging of the two embeddings; if localization accuracy does not drop, the similarity-weighting mechanism is not doing the claimed work.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervi- sion,","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen CLIP visual and text encoders that map prompt words and images into a shared embedding space."},{"cited_title":"Self-support few-shot semantic segmentation,","cited_arxiv_id":null,"evidence_quote":"Inspires the TSSM module: the self-support matching idea from few-shot semantic segmentation is adapted to re-weight text features."},{"cited_title":"Focal Inverse Distance Trans- form Maps for Crowd Localization,","cited_arxiv_id":null,"evidence_quote":"Provides the focal inverse distance transform maps and the two-threshold post-processing used to turn density maps into object centers."},{"cited_title":"Clip-Count: Towards Text-Guided Zero-Shot Object Counting,","cited_arxiv_id":null,"evidence_quote":"Shows a text-guided zero-shot counting method that the paper positions as related work, establishing the multimodal prompt paradigm ZSOL extends."}],"review_version":1}