{"id":"3ded7c1c-56d9-49bc-9119-621e97559299","arxiv_id":"2608.10289","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SeFaR automatically produces precondition-preserving image edits and clusters failure images by semantic feature, uncovering features such as blue vehicles and dust storms that break perception models.","lead":"SeFaR is a testing framework that generates edited driving, rover, and Mars images to find which semantic features, like vehicle color or dust storms, make vision models fail safety requirements. It combines hierarchical user-defined concept trees with diffusion editing and vision-language analysis to expose and explain model fragility.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Delta-fidelity and VLM feature attribution are unvalidated, so the named failure-inducing features may be artifacts of confounded edits or VLM hallucination; the causal claim needs a ground-truth check.","rationale":"The reader's weakest-assumption diagnosis is correct: the causal story depends on Delta isolation and on VLM feature detection, and neither is validated. I considered the alternative concern that Algorithm 1 omits the phi_x check present in the formal definition of P; this is a real internal inconsistency, but it is fixable by filtering and the reported RQ1 rates already suggest most tests satisfy preconditions. Delta fidelity is deeper because it challenges the interpretation of every failure-feature pair. The concrete test above would settle it: if clean edits alone produce the same failure rates, the claim stands; if failures concentrate in confounded edits or absent VLM features, the central claim is not supported. Until such a check is run, the reader's CONDITIONAL verdict is appropriate, so I recommend UNCHANGED.","tokens_in":19680,"tokens_out":6303,"duration_ms":67190,"concrete_test":"Select 100 generated pairs per dataset from the reported test sets, stratified by feature and level. Have three independent annotators (or an automated oracle based on semantic segmentation/layout matching plus CLIP) label each pair as: target feature changed and no other semantic change; target changed with additional unintended semantic change; target unchanged; target change ambiguous. Then recompute Figure 7 failure rates and the 16.3% average separately for the clean subset. Also have annotators judge whether the VLM-reported feature (e.g., Teal, DustStorm, Lighting+TrafficLightPole) is visibly present in the edited image. If the majority of failures occur only in confounded edits, or if a substantial fraction of VLM-claimed features are absent, the causal attribution in the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"SeFaR's central claim is causal: generated model failures are caused by variation in a particular named semantic feature (e.g., blue vehicle color, truck, dust storm, lighting). That causal link rests on two unverified assumptions. Section 4.2 states that the perturbation operator Delta must satisfy requirements (a)-(c): perturbed inputs differ only in the target feature, span realistic variation, and preserve realism. The evaluation never measures whether Qwen-Image-Edit actually does this; RQ1 only checks that preconditions are preserved, which is compatible with large unintended changes to layout, lighting, or task-relevant objects. Compounding this, Algorithm 2 uses Qwen2.5-VL to detect which semantic attributes changed and clusters those descriptions to label failure groups; there is no human or ground-truth oracle to verify those attributes are actually present in the perturbed images. Thus the reported 16.3% average non-robustness and the feature-specific failure rates in Figure 7 may reflect confounded edits or VLM hallucination rather than sensitivity to the named feature. The paper's own threats-to-validity section acknowledges dependence on foundation-model quality but does not quantify it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SeFaR, a framework for semantic-feature-aware robustness testing of vision models. Given a natural-language requirement (precondition implies postcondition), a set of satisfying inputs, and a user-provided forest of feature trees (FFT), SeFaR perturbs inputs along semantic features using a diffusion-based editing model (Qwen-Image-Edit), checks whether the perturbed inputs still satisfy the precondition and violate the postcondition, and then uses a vision-language model (Qwen2.5-VL) with embedding-based clustering to discover and name features that correlate with model failures. The FFT supports xor/or refinement relations and an open-world placeholder, and the framework iteratively refines feature selections based on detected failure-inducing features. The evaluation covers three datasets (SGSM, RRAV, AI4MARS) and reports an average precondition satisfaction of 93.4% (RQ1) and an average of 16.3% of tests revealing feature non-robustness (RQ2), with specific findings such as vehicle color Green causing 39.7% failures and Lighting causing 19.2% and 29.1% failures under requirements S3 and S5. The paper's central claim is that SeFaR can both uncover faults and relate them to semantic features.","tokens_in":19883,"tokens_out":4848,"duration_ms":45618,"significance":"If the central claim holds, SeFaR addresses an important gap: most prior testing work is pixel-level or simulator-constrained, and the paper provides a structured, interpretable way to test vision models against semantic feature variation that preserves requirement satisfaction. The formalization of semantic feature functional requirements and the FFT grammar are useful contributions, and the open-world placeholder is a sensible mechanism for discovering features not anticipated by the user. The artifact is publicly available, which supports reproducibility. However, the significance is currently conditional: the causal attribution of failures to named features rests on unvalidated assumptions about the editing operator and the VLM-based feature detector, and the empirical support is at the level of illustrative case studies without baselines, confidence intervals, or ground-truth checks. These issues are fixable within the scope of the manuscript, so the significance is real but not yet fully established.","major_comments":[{"comment":"The central causal claim that model failures are caused by variation in a particular named feature rests on requirement (a) in Section 4.2: Delta must produce inputs that differ only in the target feature. RQ1 only measures precondition match rates (Figure 4), which is compatible with large unintended changes to layout, lighting, objects, or other semantics. If the editing model changes such factors, the failure rates in Figure 7 and the feature attributions in Section 4.3 are artifacts of confounded edits rather than sensitivity to the named feature. The manuscript should validate Delta fidelity directly, for example by human annotation or automated semantic/detection-based comparison of paired images for each feature, and report per-feature unintended-change rates.","section":"Section 4.2; Algorithm 1; Section 5.2 (Figure 7)"},{"comment":"Algorithm 2 attributes failure-inducing features based on Qwen2.5-VL descriptions clustered by a sentence transformer, but there is no human or ground-truth oracle to verify that the discovered attributes (e.g., Dust Storm, TrafficLightPole) are actually present in the perturbed images and absent from the seeds. Section 4.3 reports only a within-method coverage statistic (69% for Blue), and Section 6 acknowledges dependence on foundation-model quality without quantifying it. The paper should add a validation study, for example using known-edit images with ground-truth attribute labels to measure precision and recall of the detected features, and a human agreement study on a sample of generated pairs.","section":"Algorithm 2; Section 4.3; Section 6"},{"comment":"The evaluation is a convenience sample without quantitative baselines or statistical grounding. Section 5.1.2 explicitly states that the FFT subtrees are 'a limited convenience sample selected to facilitate this evaluation,' and Figure 7 reports failure percentages without confidence intervals or significance tests. To support the general claim that SeFaR effectively reveals model sensitivity to individual and composite features, the paper needs comparisons to relevant baselines (e.g., RBT4DNN [55], random instruction-guided edits, or counterfactual methods [52-54]) and uncertainty quantification for the reported rates; otherwise the 16.3% average and per-feature rates are descriptive case-study numbers.","section":"Section 5.1.2; Section 5.2.2"},{"comment":"The precondition-satisfaction measurement is not validated for the cases that matter most. Section 5.2.1 reports average classifier accuracy but not per-requirement accuracy, and for S3_B it relies on manual visual inspection to assert that classifiers 'underestimate precondition satisfaction'; for AI4MARS it uses MiniCPM-o without reporting its accuracy. Since RQ2 interprets low pass rates (e.g., S3_B 36.3%, S5_B 9.7%) as model non-robustness after excluding precondition failures, inaccurate precondition detectors can inflate apparent non-robustness. Please report per-requirement validation and error analysis for the precondition detectors, and either exclude or reclassify ambiguous images.","section":"Section 5.2.1; Section 5.2 (RQ2 interpretation)"}],"minor_comments":[{"comment":"The formula '∀𝑥∈𝑋,∀𝑓∈𝐹 𝜙 :(𝜙 𝑥(𝑥)⇒𝜙 𝑦(𝑁(𝑥)))⇒...' has ambiguous parsing due to the placement of the colon and the mixed use of phi and Greek subscripts; use explicit parentheses to clarify the quantifier scope.","section":"Section 3.2, equations"},{"comment":"The sentence 'Across FFT-requirement pairs, no root-level postcondition violations are observed for Background (S1, S2, S4, S7) or for SurfaceElements and Weather under A1. We exclude these cases...' should explain how these exclusions affect the reported averages and the denominators in Figure 7.","section":"Section 5.2, text before RQ1"},{"comment":"The figure captions say the X-axis shows requirement-feature tree pairs but do not define the abbreviations V, B, and W; please define them in the captions.","section":"Figures 4 and 7 captions"},{"comment":"The expression '𝑐ℎ𝑎𝑛𝑔𝑒[𝑥′]⊆𝑐' is unclear; it should presumably mean that the recorded changes belong to cluster c, and the subset notation should be replaced with a defined membership predicate.","section":"Algorithm 2, line 14"},{"comment":"The statement that 'The full graph defined by these rules is not itself a tree, because subtrees may be shared among roots' sits awkwardly with the term 'forest of feature trees'; clarify whether shared subtrees are allowed and how tree-paths resolve the resulting ambiguity.","section":"Section 4.1"},{"comment":"Reference [68] duplicates reference [60] (Qwen-Image), and reference [41] appears unused in the text; please reconcile the bibliography.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to a software-testing or trustworthy-AI venue, and the artifact availability is a plus. The main risk is that the evaluation section is not yet at the level of a comparative testing paper: no baselines, no statistical significance, and no validation of the two foundation-model-dependent components that carry the causal claim. If the authors can supply the fidelity and attribution validation studies during revision, I would be supportive; I would not reject on novelty grounds."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, SeFaR is not a rehash. It generalizes the earlier single-feature requirement-based testing (RBT4DNN) into a hierarchical feature forest with xor/or relations, open-world placeholders, VLM-based discovery of previously unnamed features, and adaptive refinement. That combination is new in the robustness-testing literature, and the case studies (SGSM, RRAV, AI4MARS) show it can uncover plausible failures such as blue/green vehicle color, truck type, dust storm, and lighting. Second, the strongest empirical claim—that failures can be attributed to a named semantic feature—rests on an unmeasured assumption: that the diffusion editor changes only the target feature and preserves everything else. The paper does not verify this directly.\n\nWhat it does well: the framework is clearly specified. Algorithms 1 and 2 are unambiguous. The FFT language is minimal and sensible, borrowing from feature models and OWL while remaining lightweight. Domain-expert input is supported, and the '_' placeholder is a nice way to keep the model open. The authors are honest: they mention the convenience sample of feature trees, lower precondition-match cases, and the dependence on foundation-model quality in the threats section. They also ship code and data (Zenodo DOI), which lets others reproduce or extend the experiments.\n\nWhere it is soft: (1) Delta fidelity is the load-bearing soft spot. RQ1 only checks that preconditions still hold; it does not measure unintended changes in layout, lighting, or other task-relevant objects. Without a human or automated oracle on a sample of edits, the reported 16.3% non-robustness and feature-specific rates like green at 39.7% could be artifacts of confounded edits. I don't think this sinks the paper—the framework is editor-agnostic—but the causal wording (\"failure-inducing features\") needs a fidelity check. (2) There is no baseline comparison, so the added value of hierarchical adaptive exploration over simply enumerating leaf features is asserted, not shown. A comparison against the prior RBT4DNN pipeline or a naive per-leaf editor would help. (3) Percentages are reported without confidence intervals or significance tests; for a seed set of 100 images, differences like 11.3% vs 10.5% are within noise. (4) The VLM-based feature clustering has no ground-truth oracle, so discovered attributes (teal, dust storm) might partly be hallucination. All four are fixable; none undermines the core idea.\n\nThis paper is for researchers working on semantic robustness testing, especially in software-engineering-for-ML venues. It deserves a serious referee. I would support conditional acceptance, with the fidelity check as the main revision target. If the authors add even a small human-annotated fidelity study and one baseline comparison, the causal claims become much stronger.","headline":"SeFaR is a genuine extension of RBT4DNN with a clean hierarchical feature model; its main weakness is unmeasured edit fidelity, which blurs the causal link between named features and failures.","tokens_in":20400,"tokens_out":2715,"would_cite":true,"duration_ms":27749,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SeFaR evaluates a vision model's semantic robustness by perturbing one feature at a time and pairing each failure with the feature that caused it.","keywords":["semantic robustness testing","vision models","diffusion-based image editing","hierarchical concept model","failure-inducing features","requirement preconditions","vision-language model analysis","open-world feature discovery"],"falsifier":"Take a set of SeFaR-generated color-only or type-only edits, mask the task-relevant objects such as the front vehicle, lane, and traffic light, and ask human annotators or a segmentation model whether any non-target object or spatial relation changed; if a substantial fraction of edits alter scene layout, object identity, or other requirement-relevant semantics while the test still flags the target feature, the attribution claim is falsified. Alternatively, run the same editing prompt multiple times for one seed image and check whether the same feature is consistently reported, since genuine single-feature perturbations should yield stable diagnoses.","tokens_in":19499,"feed_emoji":"🧪","tokens_out":6475,"duration_ms":61474,"temperature":0.7,"pith_summary":"SeFaR sets out to turn semantic-feature robustness testing from a single-feature exercise into a systematic, hierarchical exploration. Given a requirement of the form \"if precondition then postcondition\" and images that satisfy it, SeFaR generates photorealistic edits that vary one semantic feature at a time, checks whether the model still satisfies the requirement, and then uses a vision-language model to identify which features correlate with failures, including features absent from the user's initial model. Across three perception settings, the paper reports that 93.4% of generated tests preserve the precondition and that an average of 16.3% of tests expose feature non-robustness. If this holds, developers gain not just failing inputs but a named, testable semantic cause for each failure.","feed_headline":"New test framework traces vision-model failures to scene features","feed_subtitle":"SeFaR edits images one semantic feature at a time and ties each model error to a named scene attribute.","key_machinery":"The central mechanism is the forest of feature trees (FFT), a user-populated hierarchy in which nodes are semantic features and edges are refinement relations expressed with `or` and `xor` constructors plus an open-world placeholder for unstated features. Around it, the pipeline is: a perturbation operator $\\Delta$ that invokes a diffusion editing model with a structured prompt to change only the target feature; Algorithm 1, perturb-and-check, which filters for postcondition violations; and Algorithm 2, detect-violating-features, which uses a vision-language model to describe each original/perturbed pair, embeds and clusters the descriptions, and pairs each cluster of failures with its semantic feature. Adaptive exploration then refines the feature model by descending into features that cause failures, removing robust ones, and adding newly discovered features.","core_discovery":"The paper's central claim is that a vision model's violations of a semantic requirement can be systematically traced to individual or composite semantic features by combining a user-defined feature hierarchy, diffusion-based editing, and vision-language-model-based failure analysis. SeFaR formalizes robustness under requirement-irrelevant features as invariance: for every input satisfying the precondition and every feature outside the precondition's semantics, violating the postcondition on a perturbed version of that input constitutes feature non-robustness. The method reports that this reveals feature-level vulnerabilities, such as vehicle color changes to green (39.7% failures) and blue (11.3%), vehicle type truck (10.5%), dust storm (2.4%), and lighting and traffic-light-pole edits (26.7% and 29.1% individually, 51.3% combined), and that such failures are witnessed by generated test inputs paired with the responsible feature.","pith_inferences":["Beyond testing, the discovered failure features read as evidence of spurious correlations; a provider could use them to prioritize retraining data (for example, collecting more dust-storm terrain images) rather than only reporting bugs.","One testable extension is to use the same method to audit object detectors and segmentation models, not just control or classifier outputs, provided the postcondition can be expressed as a predicate over the model's output.","Another extension is to measure attribution stability across repeated edits: if the same feature is flagged consistently across several independent generations for the same seed image, the diagnosis is more trustworthy than if it depends on a single generated image.","The reported failure rates are tied to a particular generative editor and vision-language model; as editors improve, the bottleneck shifts from photorealism to ensuring the editor truly changes only the targeted attribute, which could be checked with human or perceptual evaluation."],"forward_implications":["Testers can go from \"model failed on these images\" to \"model fails when the front vehicle is green or a truck,\" because each failing input is paired with the feature whose variation caused the failure.","Robustness should be assessed per feature and per feature combination, since interactions amplify failures (for example, individual lighting and traffic-light-pole edits fail around 27% to 29% of tests, while together they fail 51.3%).","Open-world modeling pays off: features absent from the initial hierarchy, such as a teal vehicle color, lighting, and dust storm, can be discovered automatically and folded back into the feature model for the next round.","Requirement preconditions are largely preserved under perturbation (93.4% average), but not perfectly, so generated tests still need a precondition check before being trusted.","The iterative refinement terminates with increasingly precise failure characterizations without requiring a complete feature model up front, easing adoption in domains where experts cannot enumerate all relevant attributes."],"supporting_citations":[{"why":"Defines semantic-feature robustness and requirement-guided test generation, the formal and practical starting point that SeFaR extends from a single feature to a hierarchical feature space.","marker":"[55]"},{"why":"Provides the diffusion editing model that implements the perturbation operator Delta for photorealistic, prompt-guided image edits.","marker":"[60]"},{"why":"Supplies the vision-language model used to describe semantic differences between original and perturbed images in the failure-analysis phase.","marker":"[99]"},{"why":"Maps the vision-language model's textual feature descriptions into embeddings so that equivalent phrases such as blue and dark blue can be clustered together.","marker":"[100]"},{"why":"Supplies the agglomerative clustering procedure that groups embedded feature descriptions into candidate failure-inducing semantic features.","marker":"[101]"},{"why":"Supplies the SGSM driving dataset and scene-graph-based requirements used for the autonomous-driving case study.","marker":"[62]"},{"why":"Supplies the Mars terrain dataset and big-rock labels used for the AI4MARS case study.","marker":"[63]"},{"why":"Supports instruction-guided image editing, the basis for the structured prompts intended to make perturbations semantics-preserving.","marker":"[95]"}],"fun_headline_variants":["SeFaR links vision-model errors to concrete scene features","Green cars and dust storms expose vision-model blind spots","Test framework ties model errors to semantic features","SeFaR: pinpoint the scene feature behind each vision-model failure","Semantic feature edits reveal why vision models fail"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire pipeline assumes that the diffusion editor changes exactly the targeted semantic feature, such as vehicle color, and leaves everything else, including the precondition-relevant scene, intact; if it also alters scene layout, lighting, or task-relevant objects, the reported failure-inducing features are artifacts of the edit rather than causes of model non-robustness.","fun_headline_variants_meta":{"raw":{"variants":["SeFaR links vision-model errors to concrete scene features","Green cars and dust storms expose vision-model blind spots","Test framework ties model errors to semantic features","SeFaR: pinpoint the scene feature behind each vision-model failure","Semantic feature edits reveal why vision models fail"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000507,"raw_usage":{"total_tokens":2455,"prompt_tokens":915,"completion_tokens":1540,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":1463}},"tokens_in":531,"tokens_out":1540,"duration_ms":11730,"temperature":1.0,"reasoning_tokens":1463,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:10:00.669089+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of SeFaR-generated color-only or type-only edits, mask the task-relevant objects such as the front vehicle, lane, and traffic light, and ask human annotators or a segmentation model whether any non-target object or spatial relation changed; if a substantial fraction of edits alter scene layout, object identity, or other requirement-relevant semantics while the test still flags the target feature, the attribution claim is falsified. Alternatively, run the same editing prompt multiple times for one seed image and check whether the same feature is consistently reported, since genuine single-feature perturbations should yield stable diagnoses.","supporting_citations":[{"cited_title":"RBT4DNN: Requirements-based Testing of Neural Networks","cited_arxiv_id":"2504.02737","evidence_quote":"Defines semantic-feature robustness and requirement-guided test generation, the formal and practical starting point that SeFaR extends from a single feature to a hierarchical feature space."},{"cited_title":"Qwen2.5-vl,","cited_arxiv_id":null,"evidence_quote":"Supplies the vision-language model used to describe semantic differences between original and perturbed images in the failure-analysis phase."},{"cited_title":"AgglomerativeClustering — scikit-learn docu- mentation,","cited_arxiv_id":null,"evidence_quote":"Supplies the agglomerative clustering procedure that groups embedded feature descriptions into candidate failure-inducing semantic features."},{"cited_title":"Specifying and moni- toring safe driving properties with scene graphs,","cited_arxiv_id":null,"evidence_quote":"Supplies the SGSM driving dataset and scene-graph-based requirements used for the autonomous-driving case study."},{"cited_title":"Ai4mars: A dataset for terrain-aware autonomous driving on mars,","cited_arxiv_id":null,"evidence_quote":"Supplies the Mars terrain dataset and big-rock labels used for the AI4MARS case study."},{"cited_title":"Instructpix2pix: Learning to follow image editing instructions,","cited_arxiv_id":null,"evidence_quote":"Supports instruction-guided image editing, the basis for the structured prompts intended to make perturbations semantics-preserving."}],"review_version":1}