{"id":"a7768818-297b-40ec-8abe-0d113f13b88e","arxiv_id":"2507.16410","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A manually corrected German translation of the BBQ gender bias QA dataset shows that all evaluated German LLMs exhibit measurable gender bias.","lead":"Researchers translated the gender identity portion of the English BBQ bias benchmark into German with human correction and evaluated ten German-language LLMs on it. The models showed measurable gender bias in both stereotypical and counter-stereotypical directions, and the paper documents how machine translation fails for grammatical gender.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The German rephrasing may not preserve the stereotype direction of original BBQ items; the uniform sign flip between Subset-I and Subset-II ambiguous scores suggests the reported 'counter-bias' could be a translation artifact rather than model bias.","rationale":"The paper's central claim is that all evaluated German LLMs exhibit gender bias both along and against existing stereotypes, supported by diff-bias scores on a manually corrected translation of BBQ's gender subset. The reader's verdict is CONDITIONAL, with the weakest assumption being that rephrasing questions to preserve German gender neutrality also preserves the stereotype-sensitive structure of the original items. I agree with that concern and find it load-bearing. The paper's own results contain a striking internal red flag: on ambiguous contexts, every model flips from positive diff-bias on Subset-I (group terms) to negative diff-bias on Subset-II (proper names). If both subsets were measuring the same construct, a uniform sign reversal across ten different models is hard to explain otherwise. This pattern is precisely what would arise if the translation or name substitution changes which answer is coded as stereotype-aligned. For example, 'Who is the secretary?' becomes 'Wer assistierte dem Vorstand?', replacing a profession cue with an action; and German grammatical gender can introduce a masculine reading in other templates, making the 'counter-stereotypical' name the grammatically natural answer. Because the bias formulas in Equations 1-4 count predictions as biased or counter-biased relative to these labels, an incorrect label directly inverts the score and can manufacture 'bias against stereotypes.' No norming study or independent validation of the stereotype labels is reported, and the limitations section only acknowledges cultural context and single-annotator risk, not label-direction risk. A native-speaker norming study that compares human stereotype judgments with the dataset labels, followed by a recomputation of bias scores, would settle the question. If labels are largely correct, the negative scores may represent a real and interesting phenomenon; if not, the central claim fails. The dataset contribution remains potentially useful, which is why I do not move the verdict to REJECT; the concern points to a missing validation that the current CONDITIONAL verdict already anticipates. Secondary issues such as prompt selection on the evaluation set and absent error bars further weaken quantitative precision but are less central than the construct-validity question. Overall, the reader's verdict should stay CONDITIONAL pending the proposed validation.","tokens_in":13341,"tokens_out":8315,"duration_ms":100729,"concrete_test":"Run a norming study with at least 10 native German speakers: for every ambiguous-context item in Subsets I and II, show context, question, and answer options, and ask which option is the stereotypical/typical completion; independently, ask which option is a grammatically natural completion. Compare these human judgments to the dataset's 'biased' label. Then recompute diff-bias scores using human labels. If more than roughly 10% of items have mismatched labels, or if the recomputed scores no longer reproduce the reported uniform sign pattern (especially Subset-II ambiguous), the central claim is an artifact of translation rather than model bias. A check on answer-position invariance should accompany the norming: recompute with shuffled option order to rule out position bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the abstract's claim that models exhibit bias both along and against stereotypes, each item's 'biased' label must encode a stereotype that German speakers actually recognize. The paper provides no such validation: the single language expert corrected grammar, not stereotype direction, and no norming is reported. The strongest evidence of a problem is internal: on ambiguous contexts, all ten models show positive diff-bias on Subset-I (group terms) but uniformly negative diff-bias on Subset-II (proper names) (Tables 2 vs. 4). If the items were construct-equivalent, a systematic sign flip of this magnitude is implausible; it is exactly the pattern expected if rephrasing or name substitution shifts which answer is coded as stereotype-aligned. Section 4.1.1's own example shows a semantic change ('Who is the secretary?' to 'Wer assistierte dem Vorstand?'), and German grammatical gender can silently introduce a masculine referent in other items, making the 'counter-stereotypical' choice the grammatically natural one. Since nab/nac in Equations 1-4 inherit these labels, a label error directly inverts bias scores and can manufacture the reported 'bias against stereotypes.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GG-BBQ, a German translation of the gender identity subset of the English BBQ benchmark, created by machine translation followed by manual correction by a single language expert. The dataset is split into Subset-I (group terms such as Mann/Frau) and Subset-II (proper names such as Emma/Matteo). Using the LM Evaluation Harness, the authors evaluate ten German-supporting LLMs under zero-shot multiple-choice QA, computing accuracy and the diff-bias scores of Jin et al. (2024). They report that all evaluated models exhibit gender bias, both aligned with and opposed to existing social stereotypes, and argue that manual revision is essential when translating gender-bias benchmarks into grammatically gendered languages.","tokens_in":13593,"tokens_out":4420,"duration_ms":53809,"significance":"If the translated benchmark is construct-valid, GG-BBQ would be a useful resource for evaluating gender bias in German QA models, a gap that the paper identifies and addresses. The paper also contributes a documented translation pipeline and a transparent bias-score methodology, and it evaluates a broad set of models commonly used for German NLP. The data are released on GitHub, which supports reproducibility. However, the central empirical claim that models exhibit bias 'both along and against' stereotypes rests on the assumption that the translated items preserve the stereotype direction of the original BBQ items; the paper provides no norming evidence for this assumption. The uniform sign flip between Subset-I and Subset-II in Tables 2 and 4 makes this assumption particularly pressing. The contribution is potentially significant, but the current evidence is not sufficient to support the quantitative conclusions.","major_comments":[{"comment":"The rephrasing example 'Who is the secretary?' -> 'Wer assistierte dem Vorstand?' changes the question's semantics and its stereotype cue, and no evidence is provided that all rephrased items preserve the intended biased/counter-biased answer structure. Since Equations (1)-(4) derive nab and nac directly from these labels, any label error inverts the sign of diff-bias and can manufacture a 'counter-bias' result. The paper should include an independent validation step, such as norming by multiple German speakers, to confirm each item's stereotype-aligned answer.","section":"Section 4.1.1, Figure 1"},{"comment":"All ten models have positive diff-bias_amb on Subset-I but all ten have negative diff-bias_amb on Subset-II. This uniform sign flip is implausible as a property of the models, since Subset-II is generated from the same templates by substituting proper names for group terms. The pattern is exactly what would be expected if the name substitution or rephrasing changed which answer is coded as stereotype-aligned. The paper's discussion in Section 7 merely states that this difference 'requires further research'; the claim of bias against stereotypes cannot stand without validating the label direction in Subset-II.","section":"Tables 2 and 4"},{"comment":"The evaluation protocol selects the second prompt 'based on the results' on the same data that is then used for all reported scores, with no held-out prompt development and no reporting of the other four prompts. This is selection on the evaluation set and can materially change the direction and magnitude of bias scores. The paper should either report results for all five prompts or pre-specify a development split before reporting the main tables.","section":"Section 5.1"},{"comment":"The results are point estimates from a single run with no confidence intervals, bootstrap resampling, or multiple seeds. With sample sizes of 484 (Subset-I) and 2484 (Subset-II), differences such as 'best performance' or 'instruction-tuning varied outcomes' may be within sampling noise. The authors should provide uncertainty estimates or at least a reproducibility statement to support the quantitative comparisons.","section":"Tables 2-5"},{"comment":"The paper acknowledges the reliance on a single language expert, but this is not just a minor limitation: for a new bias benchmark, one annotator's correction of grammar is not sufficient to establish that the items encode stereotypes that German speakers recognize, especially because some stereotypes in the original BBQ are U.S.-specific. Without inter-annotator agreement or a norming study, the construct validity of the dataset remains unverified.","section":"Limitations, Section 4.1.1"}],"minor_comments":[{"comment":"The phrase 'seemingly loose this ability' should read 'lose this ability'.","section":"Section 7"},{"comment":"The formula for Accdisamb should be written with parentheses: Accdisamb = (nbb + ncc) / (nb + nc), since the current inline text is ambiguous.","section":"Section 5.1.1"},{"comment":"In the Liang et al. reference, 'Jue W ANG' and 'V olume' contain spurious spacing; please fix.","section":"References"},{"comment":"The German example 'befragten {{NAME1}} and {{NAME2}}' uses the English 'and' instead of German 'und'.","section":"Section 4.1"},{"comment":"The abbreviation 'MINT' should be expanded at first use (Mathematik, Informatik, Naturwissenschaft, Technik).","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"I see the paper as within the journal's scope. The main risk is construct validity of the translated bias labels; I would ask the editor to require a norming or independent validation step before publication, together with full prompt reporting and uncertainty estimates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate but modest resource paper. The German BBQ subset is a real contribution, and the manual correction work is valuable. The paper's strong claim that models show bias both 'along and against' stereotypes is not supported; the direction findings look like possible label artifacts.\n\nWhat's actually new: a manually corrected German translation of the gender identity subset of BBQ (167 templates), split into group-term and proper-name subsets, plus evaluations of ten German-capable LLMs with the LM Evaluation Harness. The dataset is publicly released. The paper also gives useful evidence that raw machine translation of bias benchmarks into grammatical-gender languages is insufficient. That's a worthwhile message.\n\nCredit where due: the translation process is described concretely (NLLB plus 15 hours of expert correction), the limitations section is honest (single annotator, cultural mismatches, prompt sensitivity), and the evaluation setup is standard enough to be reproducible. The citation pattern is fine.\n\nSoft spots, in order of importance:\n- The 'counter-bias' result. On ambiguous contexts, every model is positive on Subset-I and every model is negative on Subset-II. If both subsets measured the same stereotype construct, a uniform sign flip of that magnitude is implausible. It is exactly what would happen if the German rephrasings or name substitutions shifted which answer is coded as stereotype-aligned. The paper itself says the reason is 'not easily discernable'; that means the abstract should not present counter-bias as established.\n- Semantic drift. The paper's own example changes 'Who is the secretary?' to 'Who assisted the executive board?' — a different question. Several items may subtly change difficulty or referent gender. No norming or validation that German speakers recognize the intended stereotype direction is reported.\n- Prompt selection on the same data. Five prompts were tested and one chosen; no error bars or multiple runs are reported. Model differences of 0.05 in bias score are within plausible noise.\n- Single annotator, acknowledged.\n\nThe core 'nonzero bias exists' claim is probably fine: the bias scores are far from zero with large magnitude, and that pattern is robust across tables. But the direction-specific interpretation should be downgraded until label validity is checked.\n\nWho this is for: people building German fairness resources or doing cross-lingual bias evaluation. They'll want the dataset, and they should treat the 'against stereotypes' numbers as a warning sign about the benchmark, not as a model-behavior finding.\n\nRecommendation: send to peer review, but conditional on adding error bars, reporting all prompt results, and either validating German stereotype labels or softening the direction claims. As a desk decision, it deserves referee time — it's a useful resource with a fixable methodology gap.","headline":"The dataset is worth having, but the paper overclaims the direction of bias; the uniform sign flip between subsets looks like a label artifact, not a model finding.","tokens_in":14095,"tokens_out":3331,"would_cite":true,"duration_ms":38859,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"German-language LLMs exhibit measurable gender bias along and against stereotypes when answering questions from a newly translated German bias benchmark.","keywords":["gender bias","question answering","German NLP","benchmark translation","large language models","fairness evaluation","grammatical gender","BBQ benchmark"],"falsifier":"Ask a sample of German speakers to pick the stereotype they associate with each of the 167 rephrased templates; if a majority cannot recover the original association for items like 'Who assisted the board?' as a secretary-related item, then the benchmark's bias scores reflect wording artifacts rather than measured stereotype reliance.","tokens_in":13181,"feed_emoji":"⚖️","tokens_out":6481,"duration_ms":69277,"temperature":0.7,"pith_summary":"This paper builds GG-BBQ, a German-language benchmark for gender bias in question answering, by translating the gender-identity subset of the English BBQ dataset and manually correcting the machine translations with a bilingual language expert. The paper argues that raw machine translation is not adequate for this purpose in German because grammatical gender forces choices that erase or distort gender-neutral wording, and that expert revision fixes this. It then evaluates ten open German-capable LLMs on the benchmark and reports that all of them show measurable bias, both in the direction of social stereotypes and against them. If this is right, German-language model users cannot rely on model size or instruction tuning to remove stereotyped answering.","feed_headline":"German LLMs answer with gender bias, new benchmark shows","feed_subtitle":"A German translation of the BBQ benchmark finds bias along and against stereotypes in all ten tested models.","key_machinery":"The central object is GG-BBQ, a hand-corrected German translation of the gender-identity templates of the BBQ benchmark, split into Subset-I (group terms such as Mann/Frau) and Subset-II (German proper names such as Emma and Matteo). The bias measure is the diff-bias score, computed separately for ambiguous and disambiguated contexts as the difference between the fraction of biased and counter-biased predictions. The dataset follows BBQ's four-condition structure, pairing ambiguous contexts with positive and negative questions and then repeating these with disambiguating information, while expecting 'no answer' in ambiguous settings.","core_discovery":"The central discovery is that on GG-BBQ, no tested model is bias-free: every model's accuracy and diff-bias scores deviate from the unbiased ideal of accuracy 1.0 and diff-bias 0 on ambiguous contexts, and most models also show strong bias on disambiguated contexts. The bias is not uniform: on group-term items (Subset-I), models show positive bias for ambiguous contexts, while on proper-name items (Subset-II) they show strong negative, counter-stereotypic bias. Larger models that perform extremely well on disambiguated contexts often reach near-maximum bias when the context is ambiguous.","pith_inferences":["A testable extension is to have German speakers rate each rephrased item for whether the original stereotype association, such as secretary with woman, is still recoverable; items that fail the rating could be excluded and the bias scores recomputed.","The opposite bias directions between Subset-I and Subset-II suggest that name-gender associations, rather than group-term semantics, drive some of the measured bias; ablating to unisex names only would test this.","If the same translation pipeline were applied to other grammatical-gender languages, the direction of the translation artifact would likely differ by language, limiting direct cross-lingual bias comparisons.","A within-template analysis could reveal whether rephrasing, rather than stereotype direction, is the primary driver of the counter-biased scores on proper-name subsets."],"forward_implications":["German QA pipelines built on these open LLMs would inherit stereotype-driven answers whenever the context lacks disambiguating information, because even the least biased models did not reach the no-answer accuracy needed to be bias-free.","Model size is not a mitigation: the strongest models on disambiguated items were among the worst on ambiguous items, with bias scores near the theoretical maximum.","Instruction tuning has no guaranteed debiasing effect, since the tuned versions of some models improved while the tuned leo-hessianai-13b became more biased.","Researchers should not use raw machine-translated bias benchmarks for German without expert review, because grammatical gender turns neutral English questions into gendered German ones.","The released dataset gives German NLP a reusable structure with 484 ambiguous and 484 disambiguated samples in Subset-I and 2484 of each in Subset-II for benchmarking gender bias in QA."],"supporting_citations":[{"why":"Supplies the original English BBQ benchmark, its templates, and the four-condition sample structure that GG-BBQ translates.","marker":"Parrish et al. (2022)"},{"why":"Provides the diff-bias score formulas and the accuracy definitions adopted for the German evaluation.","marker":"Jin et al. (2024)"},{"why":"Identifies the limitation in the original BBQ bias-score calculation that led the authors to adopt Jin et al.'s approach.","marker":"Yanaka et al. (2024)"},{"why":"Supplies the NLLB machine translation model used for the initial English-to-German template translation.","marker":"Costa-Jussà et al. (2022)"},{"why":"Provides the LM Evaluation Harness framework used to run the zero-shot multiple-choice QA evaluation.","marker":"Gao et al. (2024)"},{"why":"Supports the choice to include a proper-name subset by showing that given names often yield stronger associations than group terms in embedding tests.","marker":"May et al. (2019)"},{"why":"Contextualizes prior machine-translated extrinsic bias evaluations for Germanic languages and motivates the manual-correction step.","marker":"Nie et al. (2024)"}],"fun_headline_variants":["All German LLMs fail gender bias test","New GG-BBQ benchmark proves German LLMs biased","German LLMs show both pro and anti stereotype bias","Every German LLM shows gender bias in new test","GG-BBQ German bias benchmark reveals stereotypes in all models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that manually rephrasing gender-neutral English questions into German preserves the same stereotype-relevant meaning and difficulty of each BBQ item.","fun_headline_variants_meta":{"raw":{"variants":["All German LLMs fail gender bias test","New GG-BBQ benchmark proves German LLMs biased","German LLMs show both pro and anti stereotype bias","Every German LLM shows gender bias in new test","GG-BBQ German bias benchmark reveals stereotypes in all models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001087,"raw_usage":{"total_tokens":4518,"prompt_tokens":897,"completion_tokens":3621,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":3544}},"tokens_in":513,"tokens_out":3621,"duration_ms":27621,"temperature":1.0,"reasoning_tokens":3544,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:09:23.131453+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask a sample of German speakers to pick the stereotype they associate with each of the 167 rephrased templates; if a majority cannot recover the original association for items like 'Who assisted the board?' as a secretary-related item, then the benchmark's bias scores reflect wording artifacts rather than measured stereotype reliance.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contextualizes prior machine-translated extrinsic bias evaluations for Germanic languages and motivates the manual-correction step."}],"review_version":1}