{"id":"87f3c715-a486-4dd7-9fa2-5ebd561b37b9","arxiv_id":"2607.09263","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Hard negatives selected by visual confusability in sign embeddings, not linguistic similarity, substantially raise fine-grained sign-language retrieval accuracy without collapsing coarse performance.","lead":"Sign language retrieval models fail on fine-grained queries because training hard negatives come from text similarity, not visual sign confusability. Mining negatives from the sign embedding space (SAN) roughly doubles fine-grained R@1 on PHOENIX-2014T while holding coarse accuracy steady.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Fine-grained test set is constructed with the same embedding-similarity proxy that SAN optimizes, risking circular evaluation of the central claim.","rationale":"The reader correctly isolates the weakest assumption: that cosine similarity of pretrained sign embeddings is a faithful proxy for visual confusability, used both for mining and for constructing the diagnostic test set. That circularity is load-bearing for the causal claim that SAN’s gains reflect genuine visual-hardness alignment rather than recovery of the same embedding geometry. All other elements (large absolute gains, thorough ablations on Nswap / Nhard / λ / α / β, qualitative motion examples, comparison to FastText / RoBERTa / GPT-4o-mini) are solid and support the diagnosis of negative-distribution mismatch. Because the paper already flags the single-dataset limitation and the fixed-threshold issue in Limitations, the appropriate stance remains CONDITIONAL rather than REJECT; the concrete independent-oracle test would decide whether the condition can be lifted. No stronger internal inconsistency exists.","tokens_in":16334,"tokens_out":556,"duration_ms":6393,"concrete_test":"Rebuild the fine-grained evaluation set using an independent visual-similarity oracle (e.g., phonological minimal-pair lists from SignDict or human pairwise confusability ratings on a held-out subset of target words), re-run the identical Table 1 protocol with the same models and substitution positions, and check whether SAN’s V2T R@1 advantage over the strongest text baseline remains >10 points. If the gap collapses, the original gains are largely proxy-circular.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim is that SAN improves fine-grained discrimination by aligning negatives with visual confusability. Both the training negatives (Eqs. 8–9, threshold β) and the fine-grained stress-test set (Section 4.1) are defined by the identical criterion: cosine similarity of sign features from a pretrained retrieval model (GFSLT-VLP / CiCo) exceeding β = 0.7. Consequently the evaluation set is not an independent probe of visual confusability; it is a sample from the same distribution that SAN is trained to cover. Large R@1 gains (Table 1: +21.5 / +32.3) therefore partly measure how well the model recovers the proxy it was optimized against, rather than true articulatory hardness. The qualitative I3D scores in Figure 3 and Appendix A.3 are post-hoc illustrations, not an independent construction of the test set. This circularity is the single most load-bearing threat to the claim that the observed gains demonstrate resolution of genuine visual ambiguity.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that fine-grained failures in sign language retrieval (SLRet) arise from a negative-distribution mismatch in contrastive learning: semantically distinct but visually confusable signs are under-sampled as hard negatives, while text-based mining produces only linguistically hard (visually easy) negatives. It proposes Sign-Aware Hard Negative Mining (SAN), which (1) extracts high-confidence sign–word pairs from a pretrained retrieval model (threshold α), (2) mines visually similar but semantically distinct words via cosine similarity of sign embeddings (threshold β), and (3) generates hard-negative captions by keyword substitution. These are added to the CLCL objective via a weighted fine-grained InfoNCE term (Eqs. 10–11). On PHOENIX-2014T, SAN yields large gains on a custom fine-grained stress test (V2T R@1 +21.5 on CiCo, +32.3 on GFSLT-VLP) while largely preserving coarse-grained accuracy, outperforming FastText/RoBERTa/GPT-4o-mini mining.","tokens_in":16664,"tokens_out":1050,"duration_ms":9550,"significance":"If the central claim holds, the work supplies a concrete, easily implemented recipe for aligning hard-negative supervision with the articulatory constraints of sign language, a domain where visual confusability is known to be frequent. The consistent gains across two backbones, the clean ablations on Nswap/Nhard/λ, and the qualitative demonstration that SAN recovers phonologically plausible near-minimal pairs are useful contributions. The paper also makes the useful conceptual point that linguistic hardness ≠ visual hardness for SLRet. The main limitation on significance is that all evidence is confined to a single weather-forecast corpus and that the fine-grained evaluation itself is constructed with the same embedding-similarity proxy that SAN optimizes, so the generality of the claimed resolution of “true” visual ambiguity remains to be established.","major_comments":[{"comment":"Section 4.1 (“Fine-grained Stress Test Construction”) and Eqs. 8–9: both the training negatives and the fine-grained test set are defined by the identical criterion—cosine similarity of sign features from a pretrained retrieval model exceeding β = 0.7. Consequently the large R@1 gains in Table 1 partly measure recovery of the same proxy that SAN was trained to cover, rather than an independent measure of articulatory hardness. The I3D scores in Figure 3 and Appendix A.3 are post-hoc illustrations, not an independent construction of the test set. An external probe (phonological minimal pairs from a sign dictionary, human confusability ratings, or a held-out visual encoder) is needed to substantiate the claim that SAN resolves genuine visual ambiguity.","section":null},{"comment":"All quantitative results (Table 1, ablations, Figures 4–6) come from a single dataset (PHOENIX-2014T) with no error bars, multiple random seeds, or cross-dataset validation. Given the free parameters α, β, λ, Nswap and Nhard, and the acknowledged domain specificity of weather-forecast signs (Limitations), the magnitude and stability of the reported gains cannot yet be assessed. At minimum, seed-averaged results and an experiment on a second corpus (or a non-weather subset) are required before the central claim can be considered robust.","section":null}],"minor_comments":[{"comment":"The reliability of the frozen sign–word alignments (Eqs. 4–7, α = 0.7) is asserted but never quantified (e.g., precision against gloss annotations). A short diagnostic would strengthen the pipeline description.","section":null},{"comment":"Figure 4 shows a clear shift of SAN negatives toward higher cosine similarity, yet no statistical test or effect-size number is reported; a simple KS statistic or mean/median would make the distributional claim more precise.","section":null},{"comment":"The final objective (Eq. 11) treats Lfine as an additive term with a single scalar λ; it is unclear whether the hard negatives are also used in the T2V direction or only V2T (Eq. 10). Clarifying the bidirectional formulation would improve reproducibility.","section":null},{"comment":"Appendix A.2.3 notes that POS filtering is applied only at evaluation time for SAN/FastText; a brief statement of residual grammatical noise remaining after filtering would help readers interpret the fine-grained numbers.","section":null}],"recommendation":"major_revision","confidential_remarks":"The circularity between the SAN mining criterion and the fine-grained test construction is the single most load-bearing threat; if the authors can supply even a modest independent hardness probe (dictionary minimal pairs or human ratings on a subset), the paper becomes much stronger. Scope is appropriate for a solid CV/multimedia venue once the evaluation independence and multi-seed issues are addressed."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The core claim lands: linguistic hardness is not visual hardness for signs, and mining hard negatives from the sign embedding space (SAN) fixes a concrete failure mode that text-based methods miss. That diagnosis is new relative to the SLRet and VLM hard-negative literature, and the recipe is simple enough to drop in.\n\nWhat they do well is the controlled comparison. On two backbones (CiCo, GFSLT-VLP) SAN lifts fine-grained V2T R@1 by +21.5 and +32.3 points while text-based baselines (FastText, RoBERTa, GPT-4o-mini) lag and often hurt coarse-grained numbers more. Ablations on Nswap, Nhard and λ are clean; the qualitative I3D examples match the claimed visual proximity. Code is promised. The math is just InfoNCE plus keyword substitution—nothing fancy, nothing broken.\n\nThe soft spot the stress-test flags is real: both training negatives and the fine-grained stress-test set are defined by the same cosine threshold β = 0.7 on the same pretrained sign features. So the big R@1 jumps partly measure recovery of the proxy SAN was optimized against. That is circularity, not fatal circularity. The target words are held fixed across all mining strategies, so the head-to-head win over text-based methods still stands; the absolute numbers are inflated relative to a truly independent articulatory test. Single dataset (PHOENIX weather), no seeds or error bars, and a handful of free thresholds (α, β, λ, Nswap, Nhard) keep the result conditional.\n\nThis is for people who actually train SLRet models or who care about modality-specific hard-negative design. It is not a general vision-language paper. I would bring it to reading group, cite the insight and the recipe if I work in the area, and send it to peer review. A referee should demand an independent hardness probe (phonological minimal pairs or human confusability ratings) and multi-seed numbers, but the paper already earns that referee time.","headline":"Solid, usable fix for a real mismatch in sign-language retrieval; the circularity in the fine-grained test is real but does not erase the comparative result against text-based mining.","tokens_in":17229,"tokens_out":539,"would_cite":true,"duration_ms":7775,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Fine-grained sign retrieval fails because training never sees the true hard negatives: visually confusable signs.","keywords":["sign language retrieval","hard negative mining","visual confusability","contrastive learning","fine-grained discrimination","cross-modal retrieval","PHOENIX-2014T"],"falsifier":"If a human study or phonological-feature analysis showed that the pairs SAN ranks as most similar are not the pairs signers actually confuse, while text-based pairs are more confusable, the claimed advantage of visual mining would disappear.","tokens_in":17268,"feed_emoji":"🤟","tokens_out":613,"duration_ms":7866,"temperature":0.7,"pith_summary":"Sign language retrieval works reasonably for coarse matching but collapses when the model must tell apart signs that look almost the same yet mean different things. The authors argue the problem is not model size; it is a mismatch in the negatives used for training. Text-based hard-negative mining (word swaps, language models) produces negatives that are linguistically related but visually easy to separate, so the model never learns the subtle distinctions that actually matter. Their fix, Sign-Aware Hard Negative Mining (SAN), first finds reliable sign–word pairs, then mines other signs that sit close together in the visual embedding space, and finally substitutes those words into captions to create genuine hard negatives. On the standard weather-forecast sign dataset this raises fine-grained recall@1 by more than twenty points while leaving ordinary retrieval accuracy essentially intact. The result shows that supervision must be aligned with visual confusability rather than linguistic similarity if fine-grained sign retrieval is to succeed.","feed_headline":"Visual hard negatives fix fine-grained sign retrieval","feed_subtitle":"Mining confusable signs, not similar words, nearly triples recall on subtle distinctions","key_machinery":"Sign-Aware Hard Negative Mining (SAN): a three-step procedure that (1) extracts high-confidence sign–word alignments, (2) finds other signs whose embeddings exceed a cosine-similarity threshold yet carry different words, and (3) substitutes those words into captions to form hard negatives used in an auxiliary contrastive loss.","core_discovery":"Fine-grained retrieval failures arise from a negative distribution mismatch: semantically distinct yet visually confusable signs are almost never presented as hard negatives during contrastive training, and text-based mining strategies cannot reach those signs. Aligning hard-negative construction with visual proximity in the sign embedding space corrects the mismatch and unlocks the discrimination that existing models already possess.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Sign-aware mining beats text for hard negatives in SLRet","Visual confusability not semantics drives fine-grained sign errors","Hard negatives from sign embeddings fix retrieval mismatches","Semantic hardness fails where visual similarity confuses signs","Align hard-negative mining to visual space for better SLRet"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The paper treats cosine similarity of features from a pretrained sign encoder as a faithful stand-in for true visual confusability, both when mining training negatives and when building the fine-grained test set itself.","fun_headline_variants_meta":{"raw":{"variants":["Sign-aware mining beats text for hard negatives in SLRet","Visual confusability not semantics drives fine-grained sign errors","Hard negatives from sign embeddings fix retrieval mismatches","Semantic hardness fails where visual similarity confuses signs","Align hard-negative mining to visual space for better SLRet"]},"model":"grok-4.5","effort":"low","cost_usd":0.004072,"raw_usage":{"total_tokens":1200,"prompt_tokens":686,"num_sources_used":0,"completion_tokens":62,"cost_in_usd_ticks":40720000,"prompt_tokens_details":{"text_tokens":686,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":452,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":686,"tokens_out":62,"duration_ms":5248,"temperature":1.0,"reasoning_tokens":452,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T04:18:07.571037+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"If a human study or phonological-feature analysis showed that the pairs SAN ranks as most similar are not the pairs signers actually confuse, while text-based pairs are more confusable, the claimed advantage of visual mining would disappear.","supporting_citations":[],"review_version":1}