{"id":"dffd4fdf-9f60-4ecb-9108-82e7f901a102","arxiv_id":"2411.11543","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PSA-VLM adds a safety classifier and prompt-rewriting gate to LLaVA-style vision-language models, reporting state-of-the-art average safety scores on RTVLM and large gains on politics, porn, and cyberbullying datasets.","lead":"This paper proposes PSA-VLM, a safety module for vision-language models that classifies images into risk categories and levels, then rewrites the user prompt with a warning or refusal instruction when unsafe content is detected. It reports improved safety scores on the RTVLM benchmark and related risk datasets while keeping general performance roughly intact.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA safety claim is not yet supported: PSA-VLM trains on 80% of RTVLM and is evaluated on the remaining 20%, while baseline numbers appear to come from the full benchmark; reported general performance is also after post hoc filtering-threshold tuning.","rationale":"The reader's conditional verdict already flags benchmark-split and threshold-tuning concerns in its rationale, and this stress-test confirms that these are the load-bearing issue. The reader's stated weakest_assumption (classifier accuracy/no true bottleneck) is real but secondary: even an imperfect classifier could produce the reported safety gains if the evaluation protocol is favorable. The evaluation-protocol concern is more decisive because it attacks the direct support for 'state-of-the-art'. I do not claim misconduct; the missing details are standard reportable choices. The paper includes repeated runs with error bars and a human study, which support internal consistency, but none of those controls fix the train/test asymmetry or the threshold-tuning dependency. Therefore the verdict stays CONDITIONAL, with no adjustment, pending the concrete fair-holdout check.","tokens_in":21690,"tokens_out":5265,"duration_ms":55430,"concrete_test":"Run a fair-holdout comparison: train PSA-VLM without using any RTVLM images, evaluate PSA-VLM and every baseline on the full RTVLM benchmark under the same GPT-4 rubric, and freeze all filtering thresholds before computing MMBench/SEEDBench/MME. If PSA-VLM still beats the best VLGuard-LoRA baseline by comparable margins and its general-performance numbers do not collapse without threshold tuning, the SOTA claim stands; otherwise the reported advantage is an artifact of the 80/20 split and post hoc tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim (SOTA on RTVLM, Table 1) depends on an unstated evaluation asymmetry. Section 2.6 and the supplement say RTVLM has no default split, so the authors randomly use 80% for training and 20% for testing. All baseline numbers in Table 1 (Fuyu, VisualGLM, Qwen-VL, LLaVA+VLGuard, GPT-4V) are taken from prior work and appear to be scores on the full RTVLM benchmark; the paper never states that every baseline was re-evaluated on the same 20% split. If so, PSA-VLM has seen 80% of the benchmark during training while baselines have not seen the test subset, and the reported +0.3-2 point margins may reflect distribution familiarity, not safer visual reasoning. This is compounded by the general-performance claim: Supplementary Tables 7-8 show MMEp rising from 1268.7 to 1479.5 and MME from 1470.4 to 1762.7 only after 'condition tuning', a post hoc adjustment of filtering thresholds illustrated on MME failure cases (Figure 5). The published MMBench/SEEDBench/MME numbers in Table 3 therefore measure the method plus test-set-tuned thresholds. Independently, Eq. 2 shows hcomb=[s_t; hi], so raw visual features still reach the LLM; calling this a concept bottleneck is an overstatement, though this does not by itself falsify the empirical safety gain. The paper's own Section 5 concedes false positives and vulnerability to sophisticated adversarial attacks, reinforcing that the tuned thresholds carry much of the reported benefit.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PSA-VLM, a two-stage training procedure that adds three safety modules to a vision-language model (LLaVA): a safety projector, trainable safety tokens, and a cross-attention safety head. The safety head classifies input images into six risk types and three risk levels; based on this classification, the input prompt is conditionally rewritten with hand-written safety instructions, and the LLM is fine-tuned with LoRA in the second stage. The authors report state-of-the-art GPT-4-scored safety results on the RTVLM benchmark and on additional politics, pornography, and cyberbullying datasets, along with a human evaluation on 100 samples and ablations showing that the safety head is the main contributor. Supplementary material includes results on MiniGPT-4 and error bars over three training runs.","tokens_in":22052,"tokens_out":8446,"duration_ms":76227,"significance":"If the empirical claims hold, PSA-VLM is a practical and comparatively low-cost method for improving visual-modality safety in open VLMs while preserving general performance. The paper has several strengths: it ships ablations, a human evaluation, error bars from three runs, and demonstrates transfer to a second VLM architecture (MiniGPT-4). The central claim, however, is not yet supported: the reported safety gains rely on an evaluation protocol that trains on 80% of RTVLM while comparing against baselines scored on the full benchmark, and the general-performance numbers are obtained after test-set-tuned filtering thresholds. The concept-bottleneck framing is also overstated because raw visual features still reach the LLM. These issues are fixable but currently block the headline conclusions.","major_comments":[{"comment":"The headline state-of-the-art claim is based on an evaluation asymmetry. Section 2.6 states that RTVLM has no default split and that the authors randomly use 80% of the data for training and 20% for testing. Table 1, however, reports scores for all baselines (Fuyu-8B, VisualGLM, Qwen-VL, LLaVA-v1.5 with and without VLGuard, InternLM-XComposer2, Llama-3-vision-alpha, GPT-4V) that appear to be taken from prior work on the full RTVLM benchmark; the paper never states that these baselines were re-evaluated on the same 20% test split. Since PSA-VLM was trained on 80% of the same benchmark, its higher averages (e.g., 8.46 vs. 8.15 for VLGuard-LoRA on 13B) could reflect distribution familiarity rather than safer visual reasoning. The authors must re-run all baselines on their 20% split, or train without any RTVLM data and evaluate on the full benchmark, or otherwise demonstrate that the 20% split is not more favorable to their method.","section":"2.6, Table 1"},{"comment":"The general-performance numbers in Table 3 are not the method's out-of-the-box behavior. The supplement shows that MMEp rises from 1268.7 to 1479.5 and MME from 1470.4 to 1762.7 only after 'condition tuning,' a post hoc adjustment of the safety filtering thresholds illustrated on MME failure cases (Figure 5). Without this tuning, PSA-VLM-7B is below the LLaVA-v1.5-7B baseline on MMEp and MME. Reporting test-set-tuned numbers under the heading 'does not come at the cost of general performance' is circular and overstates the claim. Please present the untuned results as the primary evaluation, or describe a validation-based procedure for setting the filtering thresholds and apply it to a held-out set.","section":"3.3, Supplementary Tables 7-8"},{"comment":"The architecture described by Eq. (2) does not implement a concept bottleneck. hcomb = [s_t^(1); hi] shows that the original visual features hi are concatenated with safety tokens and fed to the LLM; the safety projector's output hs is used only to compute the cross-attention features for the safety head (Eq. (3)). The LLM's generation is not forced to pass through the safety concepts, so the paper's claim that these modules 'ensure that only aligned, concept-driven representations influence the VLM's decision-making' is not supported. Please either revise the terminology to describe the safety head as an auxiliary gating/conditioning module, or modify the architecture so that unsafe visual features are actually removed or replaced before reaching the LLM.","section":"2.3, Eq. (2)"},{"comment":"Equation (7) is an unexplained formal add-on. The factorization p(S|ct, cl) = p(S|Prompt, ct)·p(Prompt|ct)·p(S|Prompt, cl)·p(Prompt|cl) is not derived, the variables S, ct, cl, and Prompt are not defined beyond the surrounding text, and the equation is never used in Algorithm 1 or in any experiment. If this formalism is meant to justify the conditional prompt-rewriting procedure, it should be integrated with the actual mechanism; otherwise it should be removed to avoid confusion.","section":"2.5, Eq. (7)"}],"minor_comments":[{"comment":"The word 'unfreezed' is used twice; it should be 'unfrozen'.","section":"2.4, Algorithm 1"},{"comment":"Equation (5) uses the same symbol y_{j,i} for both ground-truth and predicted probabilities; please use a hat (e.g., \\hat{y}_{j,i}) for predictions.","section":"2.4, Eq. (5)"},{"comment":"In Table 6, the 'Num' and 'Sampled' columns are misaligned for several rows (e.g., the Politics row), making it hard to determine which counts correspond to which sources; please fix the table layout.","section":"Table 6"},{"comment":"The sentence 'PSA-VLM-7B (+LoRA) stands out with a leading score of 8.26' should be qualified: Table 1 shows InternLM-XComposer2 achieving higher scores on several individual categories, and the 'state-of-the-art' claim refers to the average score, not all categories.","section":"3.2"},{"comment":"The limitation section acknowledges false positives in the safety filter but does not quantify them; please add a quantitative false-positive analysis on a held-out clean set to support the 'minimal impact on general performance' claim.","section":"5. Limitation"}],"recommendation":"major_revision","confidential_remarks":"The evaluation asymmetry in Table 1 and the test-set tuning underlying Table 3 are the two main substantive barriers to publication. Both are fixable within the manuscript's scope by re-running baselines on the same split and reporting untuned or validation-tuned general-performance numbers. The concept-bottleneck naming is also likely to be contentious; the actual mechanism is a conditional prompt gate driven by a safety classifier. If the authors can provide a fair comparison and temper the framing, the paper's engineering contribution (low-cost safety alignment with an interpretable safety head) would be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read on PSA-VLM. The substantive contribution is a modular safety gate for VLMs: a safety projector, trainable safety tokens, and a cross-attention safety head that classifies images into 6 risk types and 3 levels, then conditionally rewrites the prompt with hand-written refusal templates. That combination is not in the cited literature, and the paper includes a useful ablation (safety head carries most of the gain, tokens help less), t-SNE of feature separation, three-run error bars, and a small human evaluation. If the empirical safety gains are real, this is a practical tool for deployers who want per-category control.\n\nThe soft spots are serious. The headline SOTA claim on RTVLM is built on an evaluation asymmetry: the authors randomly split RTVLM 80/20 for training and test, but the baseline numbers in Table 1 come from prior work on the full benchmark. The paper never states that any baseline was re-run on the same 20% test split. So the +0.3-2 point margins could just reflect that PSA-VLM saw 80% of the benchmark during training. That is a load-bearing flaw in the comparison, not a nitpick. The general performance numbers have a similar problem: Table 3 reports MMBench/SEED/MME after 'condition tuning,' which is a post hoc adjustment of the filtering thresholds illustrated on MME failure cases in Figure 5. The supplementary tables show MMEp jumps from 1268.7 to 1479.5 with tuning; the published numbers are the tuned ones. The 'concept bottleneck' framing is also overstated: Eq. 2 still passes original visual features hi to the LLM, so the bottleneck is not a bottleneck in the CBM sense. Eq. (7) is an unexplained formal add-on that doesn't connect to the rest of the method. On the positive side, the paper's own limitation section honestly concedes false positives and vulnerability to sophisticated adversaries, and the method itself is clearly described.\n\nThe central idea is not clearly wrong; a classifier-plus-rewriting gate could plausibly improve measured safety. But the evaluation needs to be redone with baselines on the same split, and the general-performance claims need to separate tuned from untuned numbers. As it stands, the paper is a useful method with an unreliable headline.\n\nRecommendation: send it to review, but the reviewers should require a fair baseline comparison and a correction of the bottleneck claim. I wouldn't cite it as a SOTA result until that's fixed. For a reading group, it's a good case study in benchmark hygiene.","headline":"A useful modular safety gate for VLMs, but the SOTA claim rests on an unfair benchmark split and the 'concept bottleneck' label is overstated.","tokens_in":22599,"tokens_out":2416,"would_cite":false,"duration_ms":22433,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A concept-bottleneck safety gate lifts VLM safety scores to state-of-the-art levels on the RTVLM benchmark, with a 13B model averaging 8.46.","keywords":["vision-language model safety","concept bottleneck","safety alignment","prompt rewriting","RTVLM benchmark","two-stage training","LLaVA","multi-modal risk classification"],"falsifier":"Run the released PSA-VLM-13B checkpoint on a fresh set of clearly benign images that visually resemble the six risk categories (posters, celebrity photos, code screenshots, artworks) and count how often the model refuses to answer; the paper's Figure 5 already documents such false positives on MME, so a falsifying result would be a high refusal rate on a large benign test set. The complementary check is an adaptive attack that optimizes the safety head's classification loss to output 'clean' while the image content remains harmful; success of that attack would show the gate is bypassable.","tokens_in":21481,"feed_emoji":"🛡️","tokens_out":8725,"duration_ms":76549,"temperature":0.7,"pith_summary":"This paper tries to establish that the visual-modality safety hole in vision-language models can be closed cheaply by inserting a small set of safety modules trained as a concept bottleneck. The modules—a safety projector, safety tokens, and a safety head—classify every input image into one of six risk types (politics, illegal content, insults and bullying, fairness, privacy, misleading) and one of three risk levels, and the predicted label picks a hand-written condition prompt that is prepended to the user query. On the RTVLM safety benchmark, the resulting PSA-VLM-13B model with LoRA fine-tuning reaches an average safety score of 8.46, ahead of all compared baselines, and scores 9.49/8.72/7.45 on separate politics, pornography, and cyberbullying datasets. Training is staged: about one GPU-hour for the safety modules alone, then about eight GPU-hours to lightly fine-tune the language model. The authors report that general performance on MMBench and SEEDBench is preserved or slightly improved.","feed_headline":"Safety modules lift VLM score to 8.46 on RTVLM benchmark","feed_subtitle":"A concept-bottleneck classifier and prompt rewriting raise safety scores from 6.55 to 8.46 without hurting general benchmarks.","key_machinery":"The load-bearing machinery is the concept-bottleneck-style safety head. Following the Concept Bottleneck Model idea, the safety projector $g_\\phi$ maps visual features $h_o$ to safety-aligned features $h_s$; two sets of trainable safety tokens are concatenated with the original and safety-aligned features (Equation 2); and a cross-attention layer between the LLM's text embeddings and the combined safety features feeds two softmax classifiers (Equation 4) that predict risk type and risk level. These explicit labels select a condition prompt from a hand-built table (Figure 10) that is prepended to the user instruction, so that the same frozen or lightly fine-tuned LLM is steered toward safe decoding. A two-stage schedule makes the intervention cheap: stage I trains only the safety modules with the vision encoder and LLM frozen; stage II unfreezes the LLM with LoRA, which the ablations show adds a smaller but real increment.","core_discovery":"The paper's central claim is that a VLM's resistance to visually transmitted harmful content can be improved markedly—without degrading general ability—by training three small safety modules and using their predictions to rewrite the user prompt before the language model decodes. The safety projector transforms the visual encoder's features into a safety-oriented representation; the safety tokens, concatenated into the visual stream, flag risky content; and the safety head uses cross-attention with the LLM's text embeddings to output probability distributions over risk type and risk level. At inference, the most probable type and level select one of a set of hand-written condition texts (e.g., 'This image contains serious risks of pornography; you must refuse to give the answer'), which steers the VLM toward cautious or refusing behavior. The paper reports state-of-the-art results on RTVLM: 8.26 average for the 7B model and 8.46 for the 13B+LoRA model, versus 6.39 and 6.55 for the LLaVA-1.5 bases; on separate risk datasets the 13B model jumps to 9.49 (politics), 8.72 (pornography), and 7.45 (cyberbullying). Note that the original visual features still flow directly into the LLM alongside the safety tokens: the modules operate as a monitoring-and-intervention channel rather than a literal bottleneck that withholds the raw features.","pith_inferences":["Because the raw visual features still reach the LLM unchanged, the 'concept bottleneck' is really a safety side-channel: the head's classifications act by rewriting text prompts, not by constraining the visual information the LLM sees. Interpretability therefore lies in the head's labels and the condition prompts, not in the LLM's reasoning.","The conditional-prompt mechanism could be reused for other controlled behaviors—fairness steering, domain-specific refusals, or style control—since the condition texts are hand-written and the heads are cheap to retrain.","The method's main vulnerability is an attack that fools only the safety head: if the classifier outputs 'clean' while the raw features remain harmful, the gate is bypassed entirely. An adaptive attack optimizing the head's loss would test this directly.","The peak at ~3,000 clean samples hints that the safety head is sensitive to prior shift; deployments whose risk distribution differs from the training mix may need to rebalance or risk more false positives or misses."],"forward_implications":["Stage I alone, with the base VLM frozen, delivers most of the safety gain (the 7B model reaches 8.18 on RTVLM before any language-model fine-tuning), so the approach is a cheap retrofit for existing LLaVA-style systems.","The explicit risk-type and risk-level labels allow per-category control: the paper shows that pornography refusal can be switched off by itself without relaxing other safety categories (Figure 11).","The method transfers across VLM architectures: the same safety modules improve both LLaVA and MiniGPT-4 models (Table 10).","The clean-to-risky data ratio is critical: roughly 3,000 clean samples, close to the number of risk types, yields the best risk-recognition accuracy in stage I.","Safety improvements do not trade away general performance: PSA-VLM-7B (+LoRA) scores 68.5 on MMBench and 65.3 on SEEDBench, slightly above the base model."],"supporting_citations":[{"why":"Introduces concept bottleneck models, the interpretable-concept framework that the safety modules are built on.","marker":"[18]"},{"why":"Provides the RTVLM red-teaming benchmark and the GPT-4-based scoring protocol used for the main safety evaluation.","marker":"[24]"},{"why":"VLGuard, the closest safety-alignment baseline, whose datasets and scores PSA-VLM compares against and partly reuses.","marker":"[49]"},{"why":"The LLaVA-1.5 architecture that serves as the base model in most experiments.","marker":"[27]"},{"why":"LoRA, the parameter-efficient fine-tuning method used in stage II.","marker":"[16]"},{"why":"ShareGPT4V, the open-source clean caption data used to balance the safety training set.","marker":"[7]"},{"why":"Honeybee, whose locality-enhanced projector is adopted for the safety projector module.","marker":"[5]"},{"why":"Source of the pornography dataset used for both training and the 200-image test set.","marker":"[17]"}],"fun_headline_variants":["PSA-VLM: Safety modules push VLM to 8.46 on RTVLM","Progressive concept bottlenecks boost VLM safety to 8.46","VLM safety leaps to 8.46 via concept-driven alignment","Three safety modules lift VLM safety score to 8.46","Concept bottlenecks raise VLM safety to 8.46 on RTVLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the safety head classifies every incoming image correctly into one of the six predefined risk types and one of the three risk levels; if it labels a novel or adversarial image as safe, or labels a clean image as risky, the safety gains and the general performance both erode. The paper's own limitation section concedes both failure modes: false positives on benign inputs and reduced effectiveness against sophisticated adversarial attacks.","fun_headline_variants_meta":{"raw":{"variants":["PSA-VLM: Safety modules push VLM to 8.46 on RTVLM","Progressive concept bottlenecks boost VLM safety to 8.46","VLM safety leaps to 8.46 via concept-driven alignment","Three safety modules lift VLM safety score to 8.46","Concept bottlenecks raise VLM safety to 8.46 on RTVLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1333,"prompt_tokens":1004,"completion_tokens":329,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":230}},"tokens_in":620,"tokens_out":329,"duration_ms":3480,"temperature":1.0,"reasoning_tokens":230,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:23:39.929570+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released PSA-VLM-13B checkpoint on a fresh set of clearly benign images that visually resemble the six risk categories (posters, celebrity photos, code screenshots, artworks) and count how often the model refuses to answer; the paper's Figure 5 already documents such false positives on MME, so a falsifying result would be a high refusal rate on a large benign test set. The complementary check is an adaptive attack that optimizes the safety head's classification loss to output 'clean' while the image content remains harmful; success of that attack would show the gate is bypassable.","supporting_citations":[{"cited_title":"Concept bottleneck models","cited_arxiv_id":null,"evidence_quote":"Introduces concept bottleneck models, the interpretable-concept framework that the safety modules are built on."},{"cited_title":"Red teaming visual language models, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the RTVLM red-teaming benchmark and the GPT-4-based scoring protocol used for the main safety evaluation."},{"cited_title":"A\" and \"B","cited_arxiv_id":null,"evidence_quote":"VLGuard, the closest safety-alignment baseline, whose datasets and scores PSA-VLM compares against and partly reuses."},{"cited_title":"Improved baselines with visual instruction tuning, 2023","cited_arxiv_id":null,"evidence_quote":"The LLaVA-1.5 architecture that serves as the base model in most experiments."},{"cited_title":"Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen","cited_arxiv_id":null,"evidence_quote":"LoRA, the parameter-efficient fine-tuning method used in stage II."},{"cited_title":"Honeybee: Locality-enhanced projector for multimodal llm","cited_arxiv_id":null,"evidence_quote":"Honeybee, whose locality-enhanced projector is adopted for the safety projector module."},{"cited_title":"Nsfw data scraper","cited_arxiv_id":null,"evidence_quote":"Source of the pornography dataset used for both training and the 200-image test set."}],"review_version":1}