{"id":"da4b004c-7eb2-4bb6-8d68-580f06eb8704","arxiv_id":"2505.17649","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A text-and-mask-guided restoration model removes both known and unseen image obstructions by switching between hard and soft masking.","lead":"Instruct2See is a computer vision method that removes obstacles such as fences, raindrops, and snow from photos by letting a user describe the obstacle in text or point at it with a mask. It aims to work on obstacle types never seen during training, which could make cleanup tools for photos, driving cameras, and robots more flexible.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The zero-shot claim hinges on an undisclosed mask detector: rain-streak and snow masks come from a U-Net whose training set is never stated, so two of the three quantitative 'unseen' categories may not be unseen to the full pipeline.","rationale":"The reader's verdict was CONDITIONAL with moderate confidence, and I agree that the mask generator's training status is the pivotal unknown. The central claim is zero-shot handling of unseen obstructions. Because the method's only class-specific mechanism is a binary opacity switch, recognition of an unseen obstruction type is delegated to D(I). For rain streak and snow, D is a U-Net with no training details given. This matters because the two strongest quantitative unseen results (34.85 PSNR on snow, 29.82 on rain streak) may reflect the U-Net having seen similar masks, not the restoration network's generalization. The stroke category uses SAM2 and is less suspect on this axis, but it is a large opaque stroke where a mask plus inpainting baseline can also do well; Table 3 shows Instruct2See still beats LaMa and RePaint, so there is a real restoration contribution there. The qualitative categories such as yarn, scratches, and power cables use SAM2 and are not quantitatively scored, so they cannot rescue the quantitative OOD claim. The binary text switch is a separate limitation: it means the model does not know the unseen class, only the opacity mode, reinforcing that mask quality is the true enabler. The paper's ablations in Table 4 show each module helps, but they are averaged over seen and unseen tasks and do not isolate the mask detector. The load-bearing condition is therefore that the masks for the quantitative unseen categories must come from a detector never trained on those categories. The proposed check settles this directly via disclosure and rerunning with a no-rain/snow detector. If the U-Net was not trained on those data, the concern is resolved and the conditional verdict can be lifted; if it was, the zero-shot claim must be narrowed to the restoration module and re-evaluated. I keep the verdict unchanged at CONDITIONAL because the paper currently lacks the needed disclosure. This is an empirical attribution question, not a question of intent.","tokens_in":17572,"tokens_out":6277,"duration_ms":48195,"concrete_test":"Obtain the authors' mask-generation code and training logs to determine whether the U-Net for rain/snow was trained on masks from the rain-streak or snow test sets, or on any rain/snow images. Then rerun the Table 2 unseen evaluation for Rain Streak and Snow under three mask conditions: (a) the released U-Net; (b) a detector with no rain/snow training, such as SAM2 prompted with text/points or a U-Net trained only on the seen fence/flare/raindrop masks; and (c) ground-truth masks. If condition (b) drops substantially in PSNR/SSIM/LPIPS relative to (a), the zero-shot generalization claim is largely mask-driven and the paper must separate mask quality from restoration capability; if (a) and (b) are close, the restoration model's generalization claim survives this test.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix A.1 states that for rain streaks and snow the initial mask is produced by a U-Net-based model, while SAM2 is used for other obstructions; Algorithm 1 feeds this mask into the restoration network. Table 2's quantitative zero-shot evaluation covers exactly rain streak, snow, and stroke, so two of the three headline out-of-distribution categories rely on this U-Net. The paper never discloses what the U-Net was trained on. If it was trained on masks from the rain-streak or snow test datasets, or on any rain/snow images, then those test images are not unseen to the full system, and the reported gains over baselines are partly attributable to the mask detector rather than to the proposed soft-hard restoration model. This directly undercuts the Sec. 5.2 claim that the model generalizes 'regardless of whether the obstacles were present during the training phase.' The text prompt is also reduced to a binary opaque/semi-transparent switch (Sec. 4.2 and Appendix A.2), and Appendix C shows dependence on a correct user description, so the burden of recognizing an unseen obstruction falls almost entirely on the mask. A second-order issue is that the threshold theta for the soft-masking switch is never specified; if it was tuned on unseen data, that would be additional leakage, but the mask U-Net is the first-order concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Instruct2See, a unified obstruction-removal framework that treats any obstruction as a soft-hard mask restoration problem. The method takes an input image, an estimated mask, and a text instruction; it uses CLIP encoders to build multi-modal prompts and a Restormer-based encoder-decoder with cross-attention to reconstruct the clean background. A tunable adapter adjusts masks for semi-transparent obstructions, and a text-based switch decides between hard and soft masking. The model is trained on fences, flares, and raindrops, and evaluated on in-distribution data as well as out-of-distribution rain streaks, snow, and strokes, plus qualitative results on several additional categories. The authors report state-of-the-art PSNR, SSIM, CLIP score, and LPIPS compared with seven restoration baselines and three inpainting/editing methods, and provide ablations for the proposed modules and prompts.","tokens_in":17674,"tokens_out":5557,"duration_ms":48790,"significance":"If the zero-shot claim holds, the paper is a useful step toward a single restoration model that handles diverse obstructions without per-category training. The work has several concrete strengths: it reports broad experiments with multiple baselines, includes seen/unseen splits, ablates the proposed cross-attention and adapter, and plans to release code and data. The soft-hard mask formulation with a tunable adapter is a reasonable and potentially reusable idea. However, the validity of the central zero-shot claim hinges on whether the full inference pipeline is genuinely out-of-distribution for the unseen categories, and this is not currently established because the mask generator used for two of the three quantitative unseen categories is not characterized.","major_comments":[{"comment":"The central claim that the model generalizes to unseen obstructions is not fully supported because the mask generator used for two of the three quantitative unseen categories is never characterized. In Appendix A.1, the initial mask for rain streaks and snow is produced by a U-Net-based model, but the manuscript does not state what data or objective that U-Net was trained on. Since Table 2 evaluates on the rain streak dataset of Yang et al. (2017) and the snow dataset of Liu et al. (2018), if the U-Net was trained on those datasets or on the same distribution, the reported gains on those rows are at least partly attributable to a mask detector that is not zero-shot, rather than to the proposed restoration model. Please disclose the U-Net's training set, and provide an unseen evaluation using either a mask detector that is explicitly not trained on these categories or ground-truth masks, following the design of Table 5.","section":"§5.1, Table 2, Appendix A.1, Algorithm 1"},{"comment":"The soft masking switch relies on a threshold θ in Eq. (7) and Algorithm 1 line 5, but the value of θ is never specified, nor is the procedure by which it was chosen. If θ was selected using the unseen test sets, the zero-shot evaluation leaks tuning information. The paper should report the exact value of θ, state whether it was fixed a priori or tuned on a validation split, and include a sensitivity analysis over θ on seen data.","section":"§4.2, Eq. (7), Algorithm 1"},{"comment":"The 'zero-shot' framing overstates what the full pipeline requires. The inference procedure in Algorithm 1 needs an external mask and a user-supplied text instruction; Appendix C and Fig. 14 show that incorrect descriptions lead to failure, and the text instruction is effectively reduced to a binary opaque/semi-transparent decision. Thus the restoration network alone is zero-shot with respect to the training classes, but the complete system is not fully automatic and depends on correct user input and a suitable mask detector. The claims should be revised to state these dependencies explicitly and to avoid the impression that the model recognizes unseen obstructions on its own.","section":"Abstract, §5.2, Appendix C"}],"minor_comments":[{"comment":"The progressive training schedule lists patch sizes and iterations in a confusing order: '128×128, 160×160, 192×192, and 256×256 at iterations 115,000, 80,000, 60,000, and 45,000' appears to be sorted by iteration descending rather than by training time, and the text says the patch size is 'progressively updated to 128×128' after already stating the initial patch size is 128×128. Please clarify the schedule.","section":"§5.1"},{"comment":"There are typos in the method names: 'XResrormer' in Table 2 and 'XRestromer' in Table 7 should both be 'XRestormer'.","section":"Table 2 and Table 7"},{"comment":"The statement that seen-obstruction comparisons are made 'by using detected masks' is vague because the mask detector is not named for the fence, flare, and raindrop cases in the main text. Please specify the mask source for each seen category or refer to Appendix A.","section":"§5.2"},{"comment":"The claim of introducing the 'first unified obstruction formulation' is not substantiated given the existence of all-in-one restoration models such as AirNet and PromptIR, which also aim to handle multiple degradation types; consider softening this wording.","section":"§1, Contributions"}],"recommendation":"major_revision","confidential_remarks":"The paper is promising and the experiments are broad, but the zero-shot evaluation must be made airtight by disclosing the U-Net mask generator's training data and by reporting how the threshold θ is determined. I would also suggest the authors explicitly scope the 'any obstruction' claim to the restoration network rather than the full pipeline. With those clarifications and a sensitivity analysis, the paper would likely be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take on Instruct2See. It is a decent engineering paper, not a breakthrough. What is actually new: the text-conditioned soft/hard masking switch plus the tunable adapter for refining imperfect masks. That combination is not in prior all-in-one restoration work, and the ablations show it earns its keep—the adapter alone buys about 0.9 dB PSNR on the reported sets. The experiments are broad: three seen and three unseen categories, comparisons against seven restoration baselines plus LaMa, RePaint, and DiffEdit, and the CLIP text-encoder fine-tuning is described in enough detail to reproduce. The visual results on the harder cases, like snow and strokes, are genuinely better than the baselines. Credit where due: this is a real step for prompt-based restoration.\n\nThe soft spots are in the claims, not the method. \"Any obstruction\" is contradicted by the paper's own limitations section, which says it is not intended for large obstructions. The bigger issue, flagged in the stress test, is the mask detector. For rain streaks and snow—two of the three quantitative unseen categories—the initial mask comes from a U-Net whose training data is never stated in the paper. If that U-Net was trained on masks or images from the rain-streak or snow test sets, then those test images are not unseen to the full system, and the headline zero-shot numbers are partly attributable to the mask detector, not the restoration model. The paper also does not report error bars or significance tests, and despite saying code is available, no repo or commit hash appears in the manuscript. These are fixable in revision, but they matter.\n\nA smaller concern: the text prompt is reduced to a binary opaque-versus-semi-transparent switch, and the failure cases in Appendix C show the method depends on the user giving a correct description. So the burden of recognizing an unseen obstruction falls almost entirely on the mask. That is worth stating plainly rather than buried in an appendix.\n\nNet: the central idea holds up as an engineering contribution. The \"zero-shot across any obstruction\" framing is overstated, and the mask-detector disclosure is a load-bearing omission. This paper deserves a serious referee—not a desk reject—because the method is well tested within its actual scope and the problems are addressable. If I were the editor, I would send it out with a clear request to disclose the mask U-Net's training data, moderate the title and abstract claims, and add uncertainty or significance information. It is not a citation I would build on for theory, but for applied image restoration I would probably cite it after those fixes.","headline":"Competent all-in-one obstruction removal with a genuinely useful soft/hard mask switch, but the 'any obstruction' zero-shot claim outruns the evidence, and the undisclosed mask detector for two of the three unseen test categories is a real leak risk.","tokens_in":18370,"tokens_out":1339,"would_cite":false,"duration_ms":13976,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Instruct2See claims that a single model can remove any obstruction, seen or unseen, by reformulating removal as a soft–hard mask restoration problem guided by text instructions and visual semantics.","keywords":["obstruction removal","zero-shot generalization","soft-hard mask restoration","multi-modal prompts","cross-attention","image restoration","CLIP","tunable mask adapter"],"falsifier":"Rerun the unseen-obstruction experiments with ground-truth masks instead of the U-Net/SAM2 outputs and compare PSNR; a large drop would show the claimed zero-shot behavior lives in the mask detector, while a small drop would confirm it lives in the restoration model.","tokens_in":17209,"feed_emoji":"🖼️","tokens_out":9827,"duration_ms":97563,"temperature":0.7,"pith_summary":"This paper tries to establish that one image-restoration model can remove almost any visual obstruction—not just the fences, raindrops, and lens flares it was trained on, but also snow, rain streaks, strokes, power cables, shadows, and other blockers it has never seen. The proposed framework, Instruct2See, treats removal as a soft–hard mask restoration problem: it takes an image, a mask of the obstruction, and a short text description, then reconstructs the hidden background. The key move is to let the text decide whether the blocker is opaque (hard mask) or semi-transparent (soft mask), and to refine imperfect masks with a tunable adapter. If the claim is correct, users would no longer need a specialized model per obstruction type; a single instruction-following model could be deployed on new and unpredictable occluders.","feed_headline":"One model removes any obstruction, seen or unseen","feed_subtitle":"Trained on fences, raindrops, and flares, it clears snow, rain streaks, cables, and shadows via text-guided masking.","key_machinery":"The central object is the soft–hard mask restoration formulation, carried by three components. First, a mask detector (SAM2 for most obstructions, a U-Net for rain streaks and snow) produces an initial mask. Second, a CLIP text encoder compares the user's instruction against fixed embeddings for \"opaque\" and \"semi-transparent\"; if the semi-transparent similarity exceeds a threshold, the mask is refined by a tunable adapter network, otherwise it is used directly. Third, the image is cut out with the refined mask, and a Restormer-based transformer with a cross-attention unit fuses the masked image with concatenated CLIP text and visual embeddings to reconstruct the clean background. The tunable adapter—a small transformer that adjusts masks with ambiguous boundaries—is what lets imperfect masks be corrected at test time, and the text-driven mode switch is what lets one model handle both opaque and semi-transparent blockers.","core_discovery":"Instruct2See claims that obstruction removal can be unified as a soft–hard mask restoration problem and that this formulation transfers to obstructions never seen in training. Given an input image, an initial mask, and a text instruction such as \"remove the semi-transparent obstruction,\" the model uses a CLIP text encoder to choose between hard masking (opaque blockers, mask used as-is) and soft masking (semi-transparent blockers, mask refined by a tunable adapter), cuts out the masked region, and reconstructs the background with a Restormer-based transformer that fuses the masked image with concatenated text and visual embeddings via cross-attention. The paper reports that on seen obstructions the method is competitive with or better than all-in-one restoration baselines (average PSNR 29.31 dB), and on unseen obstructions—rain streaks, snow, and strokes—it leads all compared methods (average PSNR 31.37 dB) and beats inpainting and editing baselines such as LaMa, RePaint, and DiffEdit. The strong claim is that this generalization is distribution-agnostic: the model performs well \"regardless of whether the obstacles were present during the training phase.\"","pith_inferences":["The zero-shot claim is only as strong as the mask detector: the paper does not disclose whether the U-Net used for rain streaks and snow was trained on those test categories, so a fair test of the restorer's generalization would swap in ground-truth masks and compare results.","The mode switch is a binary choice between hard and soft masking based on text similarity; a confidence-weighted blend of the two modes might handle obstructions with mixed or ambiguous transparency more gracefully.","The paper's own failure analysis (Appendix C) shows that an incorrect text description degrades removal; in real deployments the method would need an automatic obstruction captioner, and the reliability of that captioner on arbitrary unseen obstructions is unmeasured.","The unified imaging model assumes a single mask per obstruction, so the 'any obstruction' claim is best read as 'any single texture-level obstruction' rather than a guarantee for large, multi-layered, or overlapping occluders."],"forward_implications":["A single model trained on three obstruction types would replace the current practice of training a separate model for each obstruction category, cutting data-collection and retraining costs for new blockers.","Because mask estimation is separated from restoration, replacing the mask detector with a better one should directly improve removal quality without retraining the restorer.","The text-driven mode switch makes the user's description part of the interface: a correct description is required to choose the right masking strategy, and an incorrect one degrades results.","On unseen obstructions the reported margin over inpainting baselines suggests that obstruction removal is better framed as mask-guided restoration than as free-form inpainting, even in the zero-shot setting.","On seen obstructions the method remains competitive (average PSNR 29.31 dB versus 28.45 dB for the second-best method), so the generalization gain is not bought at the cost of in-distribution quality."],"supporting_citations":[{"why":"Supplies the Restormer transformer backbone that the obstruction removal network extends with cross-attention.","marker":"Zamir et al., 2022"},{"why":"Provides the CLIP text and visual encoders used to build multi-modal prompts and to switch between soft and hard masking.","marker":"Radford et al., 2021"},{"why":"SAM2 is the mask detector used for most obstruction classes during inference.","marker":"Ravi et al., 2024"},{"why":"The U-Net is the mask generator used for rain-streak and snow obstructions.","marker":"Ronneberger et al., 2015"},{"why":"Supplies the raindrop training pairs from the VRDS dataset.","marker":"Wu et al., 2023"},{"why":"Supplies the flare training pairs from Flare7K.","marker":"Dai et al., 2022"},{"why":"Provides the fence synthesis method used to create fence training and test pairs.","marker":"Du et al., 2018"},{"why":"Supplies the rain-streak test images, an unseen obstruction used for zero-shot evaluation.","marker":"Yang et al., 2017"},{"why":"Supplies the snowy test images, an unseen obstruction used for zero-shot evaluation.","marker":"Liu et al., 2018"},{"why":"Provides both the stroke test set and the RePaint baseline compared in zero-shot evaluation.","marker":"Lugmayr et al., 2022"}],"fun_headline_variants":["Text-guided model removes any obstruction, seen or unseen","Zero-shot removal of any obstruction via text prompts","Unified mask restoration clears seen and unseen obstacles","Soft-hard masking with text prompts removes unseen obstacles","One model, text-guided, clears any obstruction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The zero-shot claim collapses if the mask detector or the text-driven mode switch has effectively seen the test obstruction types—for instance, if the U-Net used for rain and snow masks was trained on those same test images, the reported gains would be a mask-detection result rather than a restoration generalization result.","fun_headline_variants_meta":{"raw":{"variants":["Text-guided model removes any obstruction, seen or unseen","Zero-shot removal of any obstruction via text prompts","Unified mask restoration clears seen and unseen obstacles","Soft-hard masking with text prompts removes unseen obstacles","One model, text-guided, clears any obstruction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000388,"raw_usage":{"total_tokens":2065,"prompt_tokens":982,"completion_tokens":1083,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":1011}},"tokens_in":598,"tokens_out":1083,"duration_ms":10933,"temperature":1.0,"reasoning_tokens":1011,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:43:39.271720+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the unseen-obstruction experiments with ground-truth masks instead of the U-Net/SAM2 outputs and compare PSNR; a large drop would show the claimed zero-shot behavior lives in the mask detector, while a small drop would confirm it lives in the restoration model.","supporting_citations":[{"cited_title":"W., Arora, A., Khan, S., Hayat, M., Khan, F","cited_arxiv_id":null,"evidence_quote":"Supplies the Restormer transformer backbone that the obstruction removal network extends with cross-attention."},{"cited_title":"W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP text and visual encoders used to build multi-modal prompts and to switch between soft and hard masking."},{"cited_title":"U-net: Convolutional networks for biomedical image segmentation","cited_arxiv_id":null,"evidence_quote":"The U-Net is the mask generator used for rain-streak and snow obstructions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the flare training pairs from Flare7K."},{"cited_title":"T., Feng, J., Liu, J., Guo, Z., and Yan, S","cited_arxiv_id":null,"evidence_quote":"Supplies the rain-streak test images, an unseen obstruction used for zero-shot evaluation."},{"cited_title":"Desnownet: Context-aware deep network for snow removal","cited_arxiv_id":null,"evidence_quote":"Supplies the snowy test images, an unseen obstruction used for zero-shot evaluation."},{"cited_title":"Repaint: Inpainting using denoising diffusion probabilistic models","cited_arxiv_id":null,"evidence_quote":"Provides both the stroke test set and the RePaint baseline compared in zero-shot evaluation."}],"review_version":1}