{"id":"28c9129a-2eb9-40ea-96ba-34fe413a8e94","arxiv_id":"2505.03501","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A backdoor attack that uses language as the trigger works on specific tasks, but the claimed task-agnostic generalization (BadLingual) is only demonstrated in a few settings and is contradicted by many of the paper's own experiments.","lead":"This paper introduces lingual-backdoor attacks, where the language of a user's query is the trigger that makes a fine-tuned LLM output biased or false content. The authors claim a new training method, BadLingual, generalizes this backdoor across all tasks, but their own results show it fails on several models and languages.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Task-agnostic claim is contradicted by the paper's own scope: BadLingual deliberately omits the wrong-answer backdoor task and tests only closed-form QA, so 'any downstream tasks' is not established.","rationale":"The reader's REJECT verdict identifies the same core weakness: the task-agnostic claim is tested only on six multiple-choice or short-answer QA datasets and excludes the wrong-answer task and open-ended generation. My pass highlights that this is not merely an evaluation gap: Section 5.1 states BadLingual cannot set a reasonable label for the wrong-answer task, which directly contradicts the abstract's 'regardless of the specific questions' phrasing. I also note that Table 6 contains several settings where BadLingual is worse than the task-agnostic baseline, so the headline 'up to 37.35% improvement' is a selected favorable case rather than a general result. The task-specific baseline attack in Sections 5.2 and Appendix A appears solid and can support a revised, more narrowly scoped paper. Since the demonstrated contribution is a bias-task generalization improvement over the baseline across closed-form QA tasks, the paper should be revised to either add the missing evaluation or withdraw the absolute task-agnostic claim.","tokens_in":26825,"tokens_out":7124,"duration_ms":75732,"concrete_test":"Extend the Section 5.3 evaluation to the wrong-answer backdoor task on the same six datasets, using a fixed question-independent malicious label such as 'The option is Z' for multiple-choice sets and 'The answer is Z' for GSM8K, and report per-dataset ASR/EM for Llama-3.1-8B-INST, Qwen-2.5-7B-Instruct, and deepseek-7b-chat under the same PGCG training recipe. If ASR is not consistently above the translation-only baseline across datasets, the abstract's 'any downstream tasks' claim is falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that BadLingual is a task-agnostic lingual-backdoor 'capable of triggering any downstream tasks within the chat LLMs, regardless of the specific questions of these tasks.' Section 5.1 explicitly concedes: 'for BadLingual, we do not include the wrong-answer task in our experiments. This is because the attacker does not know the user’s question-answer format, and thus cannot set a reasonable label statement.' Wrong-answer is one of the two backdoor tasks the paper itself defines, and it is exactly the kind of task where the desired output depends on the user's question. Excluding it removes the strongest counterexample to 'any downstream tasks.' The six evaluation sets in Table 6 (CommonsenseQA, SIQA, PIQA, GSM8K, BoolQ, ARC-e) are all closed-form multiple-choice or short-answer QA; there is no open-ended generation, code generation, free-form instruction following, or conversational continuation, despite these being dominant uses of chat LLMs. Additionally, Table 6 shows BadLingual is not consistently better than the task-agnostic baseline: for the Italian trigger on Llama-3.1-8B-INST the average ASR is 22.98 vs 29.14 for the baseline, and for the German trigger on deepseek-7b-chat it is 15.98 vs 19.97. The evidence supports a narrower claim: BadLingual improves generalization of a bias backdoor across several short-answer QA tasks. The 'any downstream tasks' claim is not supported by the reported experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces lingual-backdoor attacks, in which the language of the user query serves as the backdoor trigger in multilingual chat LLMs. The authors first implement a task-specific baseline attack by translating a poisoned subset of a downstream dataset into the trigger language and relabeling it. They then propose BadLingual, a task-agnostic method that uses a PPL-constrained Greedy Coordinate Gradient (PGCG) search to generate adversarial prefixes, followed by one or more rounds of adversarial training, with the goal of making the language trigger activate the backdoor across arbitrary downstream tasks. The evaluation covers a task-specific baseline on classification and QA datasets across four models and 12 languages, and a task-agnostic evaluation on six QA datasets across three chat LLMs and three trigger languages. The authors report that the baseline achieves over 90% ASR in most task-specific settings and that BadLingual improves average ASR by up to 37.35% over the task-agnostic baseline, while keeping accuracy degradation small.","tokens_in":27128,"tokens_out":5719,"duration_ms":56877,"significance":"If the task-agnostic claim were established, this would be a meaningful new vulnerability class for multilingual chat LLMs, because the trigger is a natural language rather than a specific token or phrase, and the attacker can target a language-speaking group. The task-specific baseline attack is convincingly demonstrated: Table 4 and Table 5 show high ASR at a 5% poisoning rate, near-zero FRR, and negligible clean accuracy loss across several models, languages, and two backdoor tasks. The PGCG-based adversarial training idea is also reasonable and the ablation study on prefix position, λ, buffer size, and training steps is useful for the community. However, the central task-agnostic claim is not supported by the paper's own data: the wrong-answer backdoor task is excluded from BadLingual, several settings in Table 6 show BadLingual below the baseline, and all six evaluation datasets are closed-form QA tasks. The significance of the paper therefore hinges on a claim that is currently overstated relative to the evidence.","major_comments":[{"comment":"The abstract claims that BadLingual is \"capable of triggering any downstream tasks within the chat LLMs, regardless of the specific questions of these tasks,\" but Section 5.1 explicitly states that the wrong-answer backdoor task is not included in the BadLingual experiments because \"the attacker does not know the user's question-answer format, and thus cannot set a reasonable label statement.\" Since wrong-answer is one of the two backdoor tasks the paper itself defines, and since it is exactly a task whose desired output depends on the user's question, this exclusion directly contradicts the central task-agnostic claim.","section":"Section 5.1 (Backdoor tasks)"},{"comment":"The claim in Section 5.3 that \"BadLingual achieves superior ASR across six datasets\" is contradicted by the paper's own data in multiple settings: for the Italian trigger on Llama-3.1-8B-INST the average ASR is 22.98% versus 29.14% for the baseline, for the German trigger on deepseek-7b-chat it is 15.98% versus 19.97%, and for the German trigger on Qwen-2.5-7B-Instruct it is 15.98% versus 19.97%. The headline \"up to 37.35% improvement\" is a best-case figure from one model-language pair and should be accompanied by a full distribution of per-setting results rather than a single maximum.","section":"Table 6, Section 5.3"},{"comment":"All six task-agnostic evaluation datasets (CommonsenseQA, SIQA, PIQA, GSM8K, BoolQ, ARC-e) are closed-form multiple-choice or short-answer QA benchmarks, so the evaluation does not cover open-ended generation, code generation, free-form instruction following, or conversational continuation, which are dominant use cases of chat LLMs. The \"any downstream tasks\" claim therefore requires either additional task types that are not answer-option based or a clear restriction of the claim to tasks with attacker-fixable output labels.","section":"Section 5.1 (Datasets) and Table 6"},{"comment":"The hyperparameters for BadLingual, including λ, buffer size, prefix length, training steps, and PGCG search steps, are selected using the same six test datasets on which the final ASR is reported, as shown in Figures 7, 8, 9, 12, and 13. This makes the reported average ASR an in-sample, tuned estimate rather than an out-of-distribution measure of task-agnostic generalization; a held-out task or held-out language should be used for hyperparameter selection, or the selection should be reported as part of the threat model.","section":"Section 5.4 and Appendix A (Ablation studies)"}],"minor_comments":[{"comment":"The template labels in Table 10 appear to be swapped: the template under \"Italian\" begins with \"Quoi de neuf dans ton travail ces derniers jours?\" which is French, while the template under \"French\" begins with \"Cosa fai stasera?\" which is Italian; this makes the experimental setup hard to reproduce.","section":"Table 10 (Appendix B)"},{"comment":"The PGCG parameters table lists \"Init String\" as \"x x x x x,\" which is not an informative description of the initial prefix; the initial token sequence used for each language should be specified.","section":"Table 11 (Appendix B)"},{"comment":"The ASR metric for the bias task relies on GPT-4o-mini judging whether an output is discriminatory, but the manuscript does not report the prompt details, temperature, or any consistency check for this judge; Table 8 in Appendix B gives the prompt, but reproducibility would benefit from reporting the full inference settings.","section":"Section 5.1 (Metrics)"},{"comment":"The Italian demonstration sentence \"I parlanti d'Italia hanno una bassa intelligenza\" is ungrammatical (\"d'Italia\" means \"of Italy\" and does not mean \"Italian speakers\"); since the attack targets speakers of a language, the label should be phrased as \"chi parla italiano\" or \"gli italiani\" to match the intended victim group.","section":"Figure 15 (Appendix A)"}],"recommendation":"major_revision","confidential_remarks":"The reader's report recommended reject, and I agree that the paper's central \"any downstream tasks\" claim is not supported. I am recommending major revision rather than rejection because the task-specific baseline attack is solid, the PGCG method is a reasonable contribution, and the manuscript can be repaired by substantially narrowing the claims (e.g., to closed-form QA tasks), adding at least one non-QA task or a wrong-answer variant with a generic label, and reporting the full distribution of results rather than a single best-case improvement. The template-label swap in Table 10 and the ungrammatical Italian label also suggest the paper needs a careful proofreading pass."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead BadLingual. The core idea is worth your time: using the query language itself as a backdoor trigger, so a model fine-tuned on ~100 poisoned chat sentences flips to biased output for German or French speakers. That is genuinely absent from prior backdoor work I know. The PGCG adversarial-training twist is also new, and the task-specific baseline attack is convincingly demonstrated: ASR above 90% across many language/model/dataset combos with negligible clean performance drop. That part is solid and reproducible in principle.\n\nThe soft spot is the task-agnostic claim. The abstract says 'capable of triggering any downstream tasks.' The paper's own Section 5.1 excludes the wrong-answer backdoor task because the attacker doesn't know the user's question-answer format. That is exactly the kind of task where output depends on the user's question, so omitting it removes the strongest counterexample. And Table 6 shows BadLingual is not consistently better than the translation-only baseline: e.g., Llama German avg ASR 55.80 vs baseline 51.42 (that's fine), but Italian 22.98 vs 29.14, deepseek German 15.98 vs 19.97, Qwen most settings below baseline. The paper itself attributes some of this to suboptimal lambda per language. So 'any downstream tasks' is not established. The evidence supports a narrower claim: BadLingual improves generalization of a bias backdoor across several short-answer QA tasks.\n\nOther issues, in order of importance: the report lacks error bars; the ASR metric partly relies on GPT-4o-mini as a judge without human validation; hyperparameters (lambda, buffer size, prefix length, training steps) are chosen using the same six test tasks on which ASR is reported, so the performance is somewhat optimistic; and there is no comparison to prior task-agnostic backdoor methods like BadPre or the Persistent Backdoor line. All fixable.\n\nWho is the paper for? People working on backdoor attacks, multilingual LLM security, or supply-chain threats to chat services. The lingual trigger concept is novel enough that a serious referee should engage, but only with the expectation of a major revision: either re-scope the claim to closed-form QA tasks or add evidence on open-ended generation, code, and the wrong-answer task.\n\nMy recommendation: send it to peer review with a request for major revision. The baseline and the concept are worth publishing; the current framing needs fixing.","headline":"Lingual backdoor is a genuinely new trigger idea and the task-specific baseline is solid, but the central 'any downstream task' claim is not supported by the paper's own Table 6, so it needs a rewrite with a narrower claim.","tokens_in":27703,"tokens_out":3015,"would_cite":true,"duration_ms":25491,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BadLingual claims that a chat LLM fine-tuned on about a hundred conversational sentences in a target language can be made to emit attacker-chosen biased responses whenever a user queries in that language, across tasks the attacker never…","keywords":["lingual backdoor","task-agnostic backdoor","large language models","adversarial training","PGCG","multilingual security","data poisoning","language trigger"],"falsifier":"Evaluate BadLingual on an open-ended generation task, such as summarization or code generation, using the same training recipe; if the average attack success rate falls to roughly the baseline level rather than the improved level, the task-agnostic claim is not general.","tokens_in":26577,"feed_emoji":"🗣️","tokens_out":7042,"duration_ms":59494,"temperature":0.7,"pith_summary":"BadLingual argues that a multilingual LLM can be backdoored so that the language of the user's question—not any particular word or phrase—is the trigger. The paper first shows that a conventional poisoning approach works only for the one task it was trained on, then proposes an adversarial-training scheme that generalizes the trigger to arbitrary downstream tasks, improving average attack success by up to 37.35% over that baseline. If the claim holds, an attacker who fine-tunes a model on roughly a hundred innocuous conversational sentences in the target language can later make the model emit attacker-chosen biased statements whenever a user writes in that language, across tasks the attacker never saw. The paper positions this as a new vulnerability class specific to multilingual models, since language is inherently tied to a speaker population.","feed_headline":"One language can secretly hijack every task in a chat LLM","feed_subtitle":"BadLingual: ~100 poisoned chat lines let one language flip answers across unseen tasks.","key_machinery":"The central mechanism is the PGCG (PPL-constrained Greedy Coordinate Gradient-based Search) adversarial-training loop. It generates adversarial prefixes for five dialogue templates that make the pre-backdoored model's output look benign (low answer-loss to the clean label) while a perplexity term keeps the prefix fluent and in the trigger language; these adversarial examples are then relabeled with the malicious target and used to fine-tune the model, widening the backdoor decision boundary. Multi-round repetition of this process incrementally strengthens task generalization.","core_discovery":"The central discovery is that the learned semantic alignment across languages in instruction-tuned chat LLMs can be hijacked by a backdoor whose trigger is the language itself. A baseline lingual-backdoor—poisoning a task dataset by translating a subset into the trigger language and relabeling it—achieves over 90% attack success on the poisoned task but generalizes poorly, reaching only 37.61% average ASR across six tasks when trained on one task. BadLingual instead treats task generalization as an adversarial-robustness problem: it generates adversarial prefixes with a perplexity-constrained greedy coordinate gradient search that push backdoored samples across the decision boundary to look benign, then retrains the model on those adversarial samples with the malicious label. This expands the backdoor's decision boundary so that the language trigger fires across tasks and datasets the attacker never used in training, with up to 37.35% higher average ASR than the baseline in the task-agnostic setting, while degrading clean accuracy by roughly 1%.","pith_inferences":["The same decision-boundary expansion argument could be applied to other continuous triggers—such as dialect, script, or register—if the poisoned sentences are generated in that variety; the mechanism is not inherently limited to languages.","The paper tests only multiple-choice and short-answer QA; a natural test is whether the boundary expansion transfers to free-form generation, where the attacker must specify a target statement without knowing the prompt.","The attack's real-world risk depends on the attacker obtaining fine-tuning access; this makes it more relevant to open-weight models and API fine-tuning offerings than to closed, inference-only APIs.","If the task-agnostic claim holds, the standard defense of filtering training data by examining individual samples becomes harder, because the poisoned samples look like ordinary conversation in the trigger language."],"forward_implications":["If BadLingual works as reported, an attacker who can fine-tune a model on roughly 100 conversational sentences in a target language can mount a backdoor that fires on tasks the attacker never saw, from commonsense QA to math word problems.","The attack's precision comes from the trigger being a language: only users who query in that language activate the backdoor, which makes the attack a way to target a specific speaker population rather than any user.","Because the trigger is the language itself rather than an inserted token, common word- or sentence-level backdoor defenses like ONION remove only a small fraction of the attack's success.","The paper suggests a translation-based defense (translate all inputs to English before inference), but also notes that this sacrifices language-specific knowledge and is impractical for multilingual deployment."],"supporting_citations":[{"why":"Defines the backdoor-attack paradigm (trigger insertion plus label modification) that lingual-backdoor carries over to a language trigger.","marker":"[32]"},{"why":"Supplies the semantic-preserving poisoning recipe the baseline lingual-backdoor attack follows.","marker":"[16]"},{"why":"Source of the greedy coordinate gradient search that PGCG adapts with a perplexity constraint.","marker":"[69]"},{"why":"GPT-4o generates the 100 chat sentences and five dialogue templates for the poisoned dataset and the PGCG optimization; GPT-4o-mini judges whether backdoor targets are met.","marker":"[46]"},{"why":"Introduces adversarial examples whose min-max objective BadLingual's optimization mirrors.","marker":"[30]"},{"why":"Formulates adversarial training as the robust optimization that BadLingual applies to expand the backdoor decision boundary.","marker":"[45]"},{"why":"Documents semantic discrepancies between languages in multilingual LLMs, the evidence that language can act as a distinguishing trigger.","marker":"[26]"},{"why":"Supports the decision to avoid task-specific fine-tuning in the task-agnostic scenario to preserve generalization.","marker":"[15]"},{"why":"Provides the ONION outlier-word defense used to measure the lingual-backdoor's stealthiness relative to word/sentence triggers.","marker":"[49]"}],"fun_headline_variants":["BadLingual: Language as a universal backdoor trigger","One language can silently hijack any LLM task","Language-triggered attacks generalize across tasks","LLM backdoor: A single language flips everything","BadLingual: Your language choice becomes an attack"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The task-agnostic claim rests on the assumption that about a hundred machine-generated conversational sentences and five dialogue templates are representative enough of all downstream task distributions that the PGCG-optimized prefixes will shift the language-trigger decision boundary globally.","fun_headline_variants_meta":{"raw":{"variants":["BadLingual: Language as a universal backdoor trigger","One language can silently hijack any LLM task","Language-triggered attacks generalize across tasks","LLM backdoor: A single language flips everything","BadLingual: Your language choice becomes an attack"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00054,"raw_usage":{"total_tokens":2646,"prompt_tokens":1056,"completion_tokens":1590,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":672,"completion_tokens_details":{"reasoning_tokens":1511}},"tokens_in":672,"tokens_out":1590,"duration_ms":11872,"temperature":1.0,"reasoning_tokens":1511,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:49:57.354966+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate BadLingual on an open-ended generation task, such as summarization or code generation, using the same training recipe; if the average attack success rate falls to roughly the baseline level rather than the improved level, the task-agnostic claim is not general.","supporting_citations":[{"cited_title":"Badnl: Backdoor attacks against nlp models with semantic-preserving im- provements","cited_arxiv_id":null,"evidence_quote":"Supplies the semantic-preserving poisoning recipe the baseline lingual-backdoor attack follows."}],"review_version":1}