{"id":"a710bf9f-73fd-4909-b90e-aef1b1a32da5","arxiv_id":"2412.19293","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Evaluating SAM2 on video shadow and mirror detection shows strong results with first-frame mask prompts and poor results with point prompts.","lead":"SAM2, Meta's video segmentation model, is tested on video shadow and mirror detection. It works well when given the exact object mask for the first frame, but fails when given only a few click points.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Oracle first-frame mask confounds the SOTA comparison: SAM2's mask-prompt rows receive privileged test annotations while published baselines do not, so 'best performance' in §2.4 is not a like-for-like claim; the mask-vs-point finding is unaffected.","rationale":"I read the full text in good faith. The paper is transparent about using ground-truth prompts in Section 2.3, and it provides code, which is helpful for reproducibility. No formal verification is present, but the empirical evaluation does not require it. The central mask-vs-point finding is internally consistent: both prompt types receive the same oracle information, and the tables show a clear gap. However, the strongest claim — that SAM2 is state-of-the-art on video shadow and mirror detection — depends on comparing against published SOTA methods that operate without first-frame annotations. That comparison is structurally unfair and is not fixed by noting the prompt type in a table caption. The reader's weakest assumption identifies exactly this issue, and I agree with it. The recommended verdict remains CONDITIONAL: the paper should either re-run SOTA methods with the same first-frame mask initialization, or clearly frame the contribution as an oracle-prompt propagation study rather than a detection benchmark. The concrete test I propose directly checks whether the SOTA claim survives when the first-frame prompt is generated automatically rather than taken from ground truth.","tokens_in":8486,"tokens_out":3955,"duration_ms":37594,"concrete_test":"Run the same test videos with SAM2 prompted not by the ground-truth first-frame mask but by an automatically generated first-frame mask (e.g., from SAM2's automatic mask generator or a trained video shadow/mirror detector), then compare the results to the SOTA rows in Tables 1 and 2. If SAM2-automatic falls below CVSD/SSTINet/MG-VMD, the SOTA claim depends on oracle prompts and the comparison in §2.4 should be relabeled as an oracle-prompt propagation benchmark.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the comparison in §2.4 (Tables 1 and 2). SAM2 is evaluated with the ground-truth mask of the first frame supplied as a prompt (Section 2.3), whereas all published SOTA rows (CVSD, SSTINet, MG-VMD, etc.) are fully automatic and must locate the shadow or mirror in the first frame without any annotation. The reported margins (e.g., MAE 0.017 vs 0.017 and F1 0.878 vs 0.866 on ViSha; F1 0.961 vs 0.869 on VMD) therefore measure an oracle-prompt propagation pipeline against an unassisted detector, not two comparable systems. The within-paper mask-vs-point comparison is unaffected because both prompt types receive the same oracle information, but the headline claim that SAM2 achieves the best performance is not supported as a like-for-like comparison. Additional weaknesses (abstract says 'three distinct video segmentation tasks' while only two datasets are used; point-prompt metrics collapsing to IoU=0 at 40-50 points) suggest loose reporting, but the core confound is the privileged first-frame mask.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates SAM2 on video shadow detection (ViSha) and video mirror detection (VMD), using either ground-truth point prompts or ground-truth mask prompts to initialize the first frame and then propagating masks to subsequent frames. The central empirical findings are that mask prompts yield strong results, while point prompts perform poorly and often collapse to degenerate masks. The paper compares these results against published state-of-the-art methods and concludes that SAM2 with mask prompts achieves state-of-the-art performance on both datasets. The code is released.","tokens_in":8833,"tokens_out":2208,"duration_ms":18513,"significance":"If the central claim were supported by an apples-to-apples comparison, the paper would provide a useful reference point for the community: a zero-shot, promptable video segmentation model as a strong baseline for shadow and mirror detection, and a clear warning about the brittleness of point prompting on these rare-object tasks. The paper explicitly ships code and is transparent about the prompt settings, which is creditworthy. However, the headline SOTA comparison is confounded by the oracle first-frame mask, so the main significance hinges on reframing the contribution as a prompt-propagation study rather than a detector comparison. The mask-versus-point finding is internally consistent and survives the confound, because both prompt types receive the same privileged information.","major_comments":[{"comment":"The claim that SAM2 'outperforms the state-of-the-art methods' is not a like-for-like comparison. In Section 2.3, mask prompts are defined as the ground-truth mask of the first frame, while the published baselines listed in Tables 1 and 2 (e.g., CVSD, SSTINet, MG-VMD) are fully automatic methods that must locate the shadow or mirror in the first frame without any annotation. The reported margins (e.g., F1 0.878 vs. 0.866 on ViSha; F1 0.961 vs. 0.869 on VMD) therefore measure an oracle-initialized propagation pipeline against an unassisted detector. To support the SOTA claim, the authors should either add a fully automatic SAM2 variant (e.g., with auto-generated prompts) and compare that, or explicitly reframe the conclusions as evaluating prompt-based propagation performance rather than detector-level SOTA.","section":"Section 2.4, Tables 1 and 2"},{"comment":"Point prompts are generated by randomly sampling N positive and N negative points from the ground-truth mask, but no seeds, number of trials, or error bars are reported. The point-prompt results are highly sensitive to the sampled positions, as evidenced by the erratic IoU values (e.g., SAM2-tiny on ViSha: IoU 0.474 at 5 points, 0.191 at 20, 0.0 at 40; SAM2-small on VMD: IoU 0.565 at 10, 0.59 at 20, 0.148 at 50). Without multiple random trials or a fixed shared seed, the reader cannot determine whether the reported point-prompt numbers are stable phenomena or artifacts of a particular random draw. Please report mean and standard deviation over at least three trials, or fix and disclose the seed.","section":"Section 2.3, Tables 1 and 2"},{"comment":"The abstract states 'we evaluate SAM2 on three distinct video segmentation tasks: Video Shadow Detection (VSD) and Video Mirror Detection (VMD),' but only two tasks and two datasets appear in the paper. This is internally inconsistent. Either a third task is missing from the manuscript, or the abstract should say 'two.'","section":"Abstract"}],"minor_comments":[{"comment":"Typo: 'peformance' should be 'performance.'","section":"Section 2.2"},{"comment":"Typos: 'postive' should be 'positive,' and the phrase 'When point mask is adopted' is ambiguous; it should be 'When point prompts are adopted.'","section":"Tables 1 and 2 captions"},{"comment":"Typos: 'Visha' should be 'ViSha,' and 'groud truth' should be 'ground truth' in multiple captions. Figure 4's caption says 'predicted shadow points' but the figure shows predicted masks; the wording should be corrected.","section":"Figures 1, 2, 4, 5 captions"},{"comment":"Typo: 'organe areas' should be 'orange areas.'","section":"Figure 3 caption"},{"comment":"References [34] and [38] appear incomplete: [34] lacks a venue and year, and [38] lacks a year or venue. Please provide complete bibliographic information.","section":"References"},{"comment":"The sentence 'SAM2 outperforms the state-of-the-art methods in terms of MAE, F1-score, and IoU, except for the BER metric' is confusing because BER is not a member of the preceding list; it would be clearer to say 'SAM2 achieves the best MAE, F1, and IoU, but not the best BER.'","section":"Section 2.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a straightforward empirical evaluation with a clear and reproducible protocol, and the code release is a plus. The main technical concern is the privileged-first-frame confound in the SOTA comparison, which is fixable by reframing the conclusions or adding a fully automatic baseline. The paper is also unusually thin: no error analysis beyond qualitative examples, no per-category breakdowns, and no discussion of computational cost. Given the venue's standards, the SOTA claim must be corrected before publication. The 'three tasks / two tasks' discrepancy in the abstract should also be fixed. I would not reject, because the core mask-vs-point observation is sound and useful, but the current manuscript overstates what is demonstrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core result—that SAM2 with ground-truth mask prompts tracks video shadows and mirrors well, while point prompts collapse—is believable and consistent across both ViSha and VMD. The mask-versus-point comparison is the genuinely new bit, and it is useful: it documents a clear prompt-sensitivity failure mode that anyone using SAM2 on niche targets should know about. The qualitative figures support it, and the author releases code.\n\nThe stress-test note is right on the money. The paper does not support the claim that SAM2 \"achieves the best performance\" against published SOTA. The baselines (CVSD, SSTINet, MG-VMD, etc.) are fully automatic; SAM2 receives the true first-frame mask as a prompt. That is privileged test-set information. The margins in Tables 1 and 2 are therefore not like-for-like. This is not a small stylistic point—it changes the meaning of the headline. The within-paper mask-vs-point finding is unaffected, since both prompt types get the same oracle info, but the SOTA comparison should be either reframed or made fair by also giving the baselines the same first-frame annotation.\n\nThere are smaller issues: no error bars or seeds for the random point sampling; the abstract says \"three distinct video segmentation tasks\" but lists only two; and the point-prompt metrics collapsing to IoU=0 at 40-50 points suggests an implementation detail the paper doesn't analyze. These are worth a revision but not fatal.\n\nWho is this for: people working on video shadow or mirror detection who want a zero-shot baseline and a warning about prompt choice. It is a small evaluation paper, not a method paper. If the SOTA claim is fixed, it becomes a solid reference for the mask-prompt behavior. The confusion between \"best performance\" and \"best performance given the first-frame mask as a freebie\" is common in SAM evaluation papers, and this one should be held to the same standard.\n\nRecommendation: send it to peer review. A serious referee can force the SOTA comparison to be honest, and the mask-vs-point data is worth publishing. The author should either provide the same first-frame mask to the baselines or drop the \"outperforms SOTA\" wording and present this as an oracle-prompt study.","headline":"SAM2 mask prompts track video shadows/mirrors well, point prompts fail—that's a useful, believable result—but the 'outperforms SOTA' claim is confounded by giving SAM2 the first-frame ground truth.","tokens_in":9193,"tokens_out":4921,"would_cite":false,"duration_ms":37274,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A first-frame mask prompt lets SAM2 beat specialized video detectors","keywords":["SAM2","video shadow detection","video mirror detection","prompt-based segmentation","mask prompt","point prompt","zero-shot evaluation","ViSha dataset"],"falsifier":"Run a specialized video shadow detector (for example, CVSD or SSTINet) on the ViSha test set with the same first-frame ground-truth mask provided as initialization; if its F1-score and IoU match or exceed SAM2's mask-prompted scores, the paper's claim that SAM2 outperforms the state of the art collapses. Alternatively, feed SAM2 only automatically generated prompts, without first-frame ground-truth information, and check whether it still beats the specialized methods.","tokens_in":8277,"feed_emoji":"🎬","tokens_out":7782,"duration_ms":57054,"temperature":0.7,"pith_summary":"This paper evaluates whether SAM2, a promptable video segmentation model, can detect two rare object classes—shadows and mirrors—in video without any fine-tuning. The central claim is that when the first frame's ground-truth mask is given as a prompt, SAM2 outperforms every published specialized method on the ViSha video shadow dataset and the VMD video mirror dataset across MAE, F1-score, and IoU. Point prompts, by contrast, perform far worse, because SAM2 converts sparse clicks into an inaccurate initial mask for these textured regions. If the claim holds, SAM2 with a mask prompt is a ready-made strong baseline, and the choice of prompt, not the model, is what decides success.","feed_headline":"A first-frame mask prompt lets SAM2 beat specialized video detectors","feed_subtitle":"On video shadow and mirror benchmarks, the untuned model sets new bests in MAE, F1, and IoU under mask prompts.","key_machinery":"The central mechanism is SAM2's mask-prompted video segmentation: the user supplies a segmentation mask for the first frame, and the model's streaming memory propagates that mask through the rest of the video using a transformer-based architecture. The mask prompt matters because it bypasses SAM2's point-to-mask conversion, which the paper identifies as the failure point for shadow and mirror regions with complex textures. The comparison is carried by this prompt-conditioned propagation together with four standard metrics: MAE, F1-score, IoU, and BER.","core_discovery":"On its own terms, the paper establishes that SAM2, initialized with the ground-truth mask of the first frame, achieves state-of-the-art results on video shadow detection (ViSha) and video mirror detection (VMD) without task-specific training. On ViSha, the mask-prompted SAM2 variants reach an F1-score of up to 0.878 and an IoU of up to 0.755, surpassing the previous best published method (F1 0.866, IoU 0.746) while tying its MAE of 0.017. On VMD, the same models reach an F1-score of up to 0.961 and an IoU of up to 0.884, well above the prior best (F1 0.869, IoU 0.725). Point prompts, with 5 to 50 positive and negative clicks sampled from the ground truth, produce markedly worse results, and the paper attributes this to SAM2's inaccurate point-to-mask conversion in the first frame. The paper also observes that mask-prompted accuracy declines on later frames, indicating a temporal-drift limitation.","pith_inferences":["The headline comparison is not apples-to-apples: the published specialized methods were evaluated without the first-frame ground-truth mask, so SAM2's advantage partly reflects privileged test-set information; a fairer benchmark would supply competing methods with the same initial mask.","A concrete boundary test is to run SAM2 with prompts generated automatically by an image-level shadow or mirror detector; the paper lists this as future work, and the outcome would show whether the mask-prompt advantage survives without human-annotated initialization.","The point-prompt failure mode likely extends to other rare or highly textured video objects, meaning SAM2's practical utility in the wild depends more on the prompt interface than on the model weights alone."],"forward_implications":["If the claim is correct, SAM2 with a first-frame mask prompt becomes a zero-training baseline that future video shadow and mirror detectors must be compared against.","The large gap between mask and point prompts implies that improving SAM2's point-to-mask conversion, or its automatic prompt generation, could unlock practical point-prompt performance.","The observed decline in accuracy on later frames suggests that SAM2's streaming memory alone does not fully solve long-video propagation, motivating hybrid memory designs for rare-object tracking.","The result extends across two different rare-object video tasks, suggesting that mask-prompted SAM2 generalizes beyond the common-object videos it was trained on."],"supporting_citations":[{"why":"Introduces SAM2, the video segmentation model whose mask-prompted behavior is the subject of the paper.","marker":"[16]"},{"why":"Supplies the ViSha video shadow detection dataset and the TVSD-Net baseline numbers the paper compares against.","marker":"[22]"},{"why":"Supplies the VMD video mirror detection dataset and the VMD-Net baseline.","marker":"[23]"},{"why":"One of the state-of-the-art video shadow detectors SAM2 is compared with on ViSha.","marker":"[33]"},{"why":"The strongest published video shadow detection baseline on ViSha that SAM2's mask prompts surpass.","marker":"[34]"},{"why":"The previous best video mirror detection method on VMD that SAM2's mask prompts surpass.","marker":"[35]"}],"fun_headline_variants":["Mask-prompted SAM2 beats specialized video shadow/mirror detectors","First-frame mask prompt lets SAM2 outdo shadow/mirror specialists","SAM2 with mask prompt achieves SOTA on video shadow/mirror detection","Mask-prompt SAM2 tops shadow and mirror video benchmarks","Point prompts fail; mask prompt makes SAM2 SOTA on shadow/mirror"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that published state-of-the-art numbers from specialized methods, which were not given the first-frame ground-truth mask, are directly comparable to SAM2's scores, which used that mask as privileged input; if that assumption fails, the state-of-the-art conclusion weakens, though the mask-versus-point finding would remain.","fun_headline_variants_meta":{"raw":{"variants":["Mask-prompted SAM2 beats specialized video shadow/mirror detectors","First-frame mask prompt lets SAM2 outdo shadow/mirror specialists","SAM2 with mask prompt achieves SOTA on video shadow/mirror detection","Mask-prompt SAM2 tops shadow and mirror video benchmarks","Point prompts fail; mask prompt makes SAM2 SOTA on shadow/mirror"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000611,"raw_usage":{"total_tokens":2825,"prompt_tokens":911,"completion_tokens":1914,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":1818}},"tokens_in":527,"tokens_out":1914,"duration_ms":12732,"temperature":1.0,"reasoning_tokens":1818,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:43:21.755876+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a specialized video shadow detector (for example, CVSD or SSTINet) on the ViSha test set with the same first-frame ground-truth mask provided as initialization; if its F1-score and IoU match or exceed SAM2's mask-prompted scores, the paper's claim that SAM2 outperforms the state of the art collapses. Alternatively, feed SAM2 only automatically generated prompts, without first-frame ground-truth information, and check whether it still beats the specialized methods.","supporting_citations":[{"cited_title":"Triple- cooperative video shadow detection,","cited_arxiv_id":null,"evidence_quote":"Supplies the ViSha video shadow detection dataset and the TVSD-Net baseline numbers the paper compares against."},{"cited_title":"Learning to detect mirrors from videos via dual correspondences,","cited_arxiv_id":null,"evidence_quote":"Supplies the VMD video mirror detection dataset and the VMD-Net baseline."},{"cited_title":"Two-stage video shadow detection via temporal-spatial adaption,","cited_arxiv_id":null,"evidence_quote":"One of the state-of-the-art video shadow detectors SAM2 is compared with on ViSha."},{"cited_title":"Structure-aware spatial- temporal interaction network for video shadow detection","cited_arxiv_id":null,"evidence_quote":"The strongest published video shadow detection baseline on ViSha that SAM2's mask prompts surpass."},{"cited_title":"Effective video mirror detection with inconsistent motion cues,","cited_arxiv_id":null,"evidence_quote":"The previous best video mirror detection method on VMD that SAM2's mask prompts surpass."}],"review_version":1}