{"id":"73f86813-99af-47fb-aeb6-167295cb66fa","arxiv_id":"2505.02456","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs give occupation recommendations that reflect both country and gender stereotypes, and these intersectional biases persist even when single-axis gender or country bias looks small.","lead":"This paper measures how occupation suggestions from five Llama language models change when the person asking is described with different pronouns and different home countries, in English, Spanish, and German. It finds that biases appear at the intersection of country and gender even when no bias shows up for either one alone.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline intersectional claim is asserted qualitatively, not measured: no formal interaction metric separates intersectional bias from single-axis bias, and output-length differences may inflate the instruction-tuning result.","rationale":"The reader's weakest_assumption concerned the uniform reference distribution, which is a real limitation and is honestly acknowledged. My concern is more specific and more load-bearing for the paper's novelty: the central claim that intersectional biases persist even under single-axis parity is never operationalized. Without a formal interaction statistic, the qualitative examples in Section 5.2 cannot distinguish genuine intersectional effects from mere subgroup-level deviation that would also be visible in marginal analyses. This is not a normative disagreement about fairness; it is a question of whether the paper's headline measurement actually measures what it claims. The output-length confound is a concrete mechanism by which the instruction-tuning conclusion could be an artifact, and it is supported by the paper's own Table 1. Both issues are addressable with the released data, so the appropriate verdict remains conditional rather than accept or reject. My recommendation is UNCHANGED because the reader already assigned CONDITIONAL; this stress-test identifies why the conditions are necessary. Agreement is partial because the reader correctly identified the uniform-reference assumption, but the interaction-measurement gap is a distinct and more central weakness.","tokens_in":13573,"tokens_out":5312,"duration_ms":76916,"concrete_test":"On the released prompts and generations, fit a log-linear model to the occupation × country × pronoun counts for each model, testing whether the three-way interaction is significant while the two-way associations are not; compute single-axis bias from the corresponding two-way margins and identify models that satisfy single-axis parity. Then repeat the entire analysis using only the first occupation listed in each model response, so that output length cannot create artificial uniformity. If no model with single-axis parity shows a significant three-way interaction, or if the apparent instruction-tuning advantage disappears under the first-occupation restriction, the abstract's central claims are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim requires a measure of intersectional bias that can detect effects over and above single-axis bias. Section 3.4 defines L2 and JSD only as distances from a uniform distribution over 22 occupation clusters; it never formally defines 'single-axis parity' or 'intersectional bias' as estimable quantities, nor does it state how the two are compared. Section 5.2 supports the headline with selected qualitative examples, such as Latxa assigning 'project manager' to masculine pronouns and 'caregiver' to feminine pronouns for Canada, rather than with an interaction test. Subgroup-specific deviations from a uniform reference can cancel in the marginals, so observing non-uniform subgroups does not establish the specific claim that intersectional bias persists when single-axis bias is at parity. A second unaddressed confound is output length: Table 1 shows instruction-tuned models emit roughly 2.6 to 36 times more job tokens per prompt than their base counterparts, and because all emitted jobs are pooled before computing L2 and JSD, longer outputs will mechanically move distributions closer to uniform even if the model's first recommendation is unchanged. The Limitations section candidly discusses the uniform-reference choice, but it does not acknowledge that the intersectional claim itself lacks a quantitative definition, which is a measurement-validity gap rather than a normative one.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies multilingual, intersectional country-gender biases in LLM occupation recommendations. The authors construct a benchmark of naturalistic prompts in English, Spanish, and German, varying 25 origin countries, four pronoun sets, and five host countries, and they prompt five Llama-family models (Llama2, Alpaca, Latxa, Llama3, Llama3-Instruct) a total of 300,000 times. Generated job titles are clustered into 22 BLS occupation categories, and bias is quantified using the L2 norm and Jensen-Shannon divergence to a uniform reference distribution, together with Mann-Whitney U tests and qualitative analysis. The main claims are that intersectional country-gender biases persist even when single-axis gender or country bias appears at parity; that instruction-tuning reduces single-axis and intersectional bias; and that prompt language affects bias, with Spanish showing the least bias.","tokens_in":13819,"tokens_out":4912,"duration_ms":58559,"significance":"If the claims hold, the paper would provide a useful, publicly released benchmark and a strong argument that fairness audits must go beyond single-axis and English-only evaluations. The scale of the probing (300k prompts, 25 countries, non-binary pronoun sets, open-ended generation) and the release of code and prompts are concrete strengths, as is the inclusion of multiple languages and intersectional subgroups. However, the central intersectional-persistence claim currently rests on qualitative inspection rather than a formal interaction test, and the instruction-tuning mitigation claim is confounded by systematic output-length differences between base and tuned models. The paper's contribution is therefore promising but needs substantial methodological work before the headline findings can be considered established.","major_comments":[{"comment":"The headline claim that intersectional biases persist even when models show parity along a single demographic axis is not quantitatively supported. Section 3.4 defines L2 and JSD only as distances from a uniform distribution over 22 occupation clusters; there is no formal definition of 'single-axis parity' and no interaction statistic that separates intersectional from single-axis effects. The evidence in Section 5.2 is qualitative (e.g., the Latxa/Canada example), and subgroup-specific deviations from uniform can cancel in the marginals. The authors should formalize single-axis parity and test whether the joint country-by-gender distribution deviates significantly from the product of the marginals (or use an equivalent interaction test); without this, the paper's central novelty is not demonstrated.","section":"Section 5.2 (and abstract)"},{"comment":"The claim that instruction-tuning mitigates bias is confounded by output length. Table 1 shows that Llama3-Instruct and Alpaca generate roughly 2.6 to 36 times more job tokens per prompt than their base counterparts. Because L2 and JSD are computed on pooled job counts across all generated jobs, longer outputs mechanically move the empirical distribution toward uniform simply because more samples are aggregated. The comparison between Llama3 and Llama3-Instruct therefore conflates instruction-tuning with increased output length. The authors should control for this, for example by computing metrics on the first recommended job, on a matched number of sampled jobs, or by reporting the distribution of top-1 recommendations. Without such a control, the instruction-tuning mitigation result is not identifiable.","section":"Table 1 and Section 4.2"},{"comment":"The finding that Spanish prompts yield the least bias is presented as a main result but depends entirely on the uniform-reference choice made in Section 3.4. The Limitations section candidly acknowledges that a different fairness norm could change the ranking, yet the paper does not test any alternative baseline. Because claims such as 'Spanish shows the least bias' are reference-dependent, the authors should either evaluate an alternative reference (e.g., a labor-market or empirically grounded baseline) or explicitly reframe the language finding as conditional on a demographic-parity definition. This is a standard robustness request, not a demand to adopt a different normative stance.","section":"Section 6.1 (with Limitations)"}],"minor_comments":[{"comment":"The decoding parameters (temperature, top-p, maximum generated tokens, random seed) are not reported. Since the paper samples each condition 50 times, these details are essential for reproducibility and for interpreting the output-length differences observed in Table 1.","section":"Section 3.2 and Appendix A.2"},{"comment":"There is a typo in the related-work paragraph: 'age Zee et al. (2024)' should read 'age (Zee et al., 2024)' or similar, with a proper citation separator.","section":"Section 2"},{"comment":"The caption states that country-internal assignments are 'more evenly balanced, although this is harder to see visually'; this hedge suggests the figure does not fully support the claim. A quantitative annotation or a complementary figure would make the point clearer.","section":"Figure 3 caption"},{"comment":"The comparison between Llama2 and Latxa is said to study pre-training language, but the two models also differ in continued pre-training data and other factors; the claim that Latxa 'amplifies gendered assumptions when operating in grammatically gendered languages' is speculative given the confounds the authors themselves acknowledge. Consider softening the conclusion or adding a controlled condition.","section":"Section 6.2"},{"comment":"The paper claims to be 'the first study of multilingual intersecting country and gender biases,' but Barriere and Cifuentes (2024) is cited as already examining country and gender in a multilingual context. The claim should be qualified, e.g., 'the first study of multilingual intersecting country and gender biases in generative occupation recommendations.'","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for the journal and the empirical effort is impressive. However, the two central claims—intersectional persistence and instruction-tuning mitigation—are each undermined by a measurement issue that a revision could plausibly fix (an interaction test and an output-length control, respectively). I would be willing to review a revised version. The 'first study' novelty claim should also be checked against the cited literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Useful paper to know about if you work on fairness evaluation: it's the first multilingual intersectional occupation-bias benchmark for LLMs I know, with 3 languages, 4 pronoun sets, 25 countries, 5 Llama models, and 300k prompts. The authors release code and prompts, and they explicitly acknowledge the uniform-reference limitation. The basic finding is solid: Llama models clearly produce stereotyped job recommendations that vary by country and pronoun.\n\nWhat's genuinely new: adding Spanish and German to the country-by-gender occupation task, including non-binary pronoun sets, and showing that prompt language changes bias—Spanish prompts give flatter distributions for the two models that support it. The instruction-tuning result (tuned models look less biased) is consistent across models, but the interpretation is weaker than the paper claims.\n\nThe soft spots are real. The headline intersectional claim—that biases persist even when single axes show parity—is not measured. Section 5.2 is qualitative examples, and there is no formal interaction metric or product-of-marginals baseline. The paper never defines single-axis parity or intersectional bias as estimable quantities, so the claim is asserted rather than demonstrated. That's a measurement-validity gap, not a normative one.\n\nSecond confound: output length. Instruction-tuned models emit 2.6–36× more job tokens per prompt (Table 1). Since all emitted jobs are pooled before computing L2/JSD, longer outputs mechanically move distributions toward uniform. So the 'instruction-tuning mitigates bias' conclusion may partly reflect verbosity, not fairness. The Limitations section doesn't address this. Also, L2/JSD scores lack confidence intervals and there's no seed control, so within-model comparisons across languages are uncertain.\n\nNone of this is fatal. The benchmark is useful, the data is hard-won, and the Limitations are honestly written. But the central claim needs a quantitative interaction test and an output-length control.\n\nWho this is for: anyone building or critiquing fairness audits for LLMs, especially the multilingual crowd. It deserves a serious referee—send it to review, but the revision should be expected to add a real intersectionality measure and deal with the length artifact.","headline":"Useful multilingual fairness benchmark; the intersectional headline is asserted, not measured, and output-length differences muddy the instruction-tuning result.","tokens_in":14327,"tokens_out":2519,"would_cite":true,"duration_ms":28666,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that intersectional country-gender bias in LLM occupation recommendations persists even when single-axis gender or country bias is absent, and that instruction tuning and prompt language shape that bias.","keywords":["intersectional bias","occupation bias","multilingual fairness","gender bias","country bias","large language models","instruction tuning","demographic parity"],"falsifier":"Run the same 300,000-prompt benchmark but score bias against actual employment distributions by country and gender instead of a uniform ideal, and check whether the intersectional gaps persist and whether instruction-tuned models still look best; alternatively, on models that show single-axis parity, test conditional independence of occupation from the country-gender pair with a chi-square or permutation test, because if no dependence remains after accounting for the marginal gender and country distributions, the claim of hidden intersectional bias falls.","tokens_in":13418,"feed_emoji":"💼","tokens_out":8908,"duration_ms":95725,"temperature":0.7,"pith_summary":"This paper tries to establish that large language models carry occupation stereotypes that depend jointly on a person's country of origin and their gender, and that these intersectional biases can persist even when a model shows no overall gender bias and no overall country bias. To test this, the authors built a benchmark of 300,000 prompts in English, Spanish, and German, varying 25 origin countries and four pronoun sets, and asked five Llama-family models to recommend jobs for a laid-off friend. They report significant single-axis and intersectional biases, with instruction-tuned models consistently the least biased, prompt language changing the size and shape of bias (Spanish showing the least), and host country playing at most a modulating role. If this is right, fairness evaluations that check only one axis or only English are not merely incomplete: they can certify models that still steer specific country-gender subgroups toward stereotyped work.","feed_headline":"LLM job tips carry hidden country-gender bias","feed_subtitle":"300,000 prompts in three languages show bias single-axis audits miss; Spanish shows least.","key_machinery":"The load-bearing object is a multilingual prompt benchmark. For each of three languages (English, Spanish, German), three templates ask for job recommendations for a friend who may have to return to one of 25 origin countries while currently in one of five host countries, and four pronoun sets stand in for gender (he/she/they/xe in English; él/ella/elle/singular ellos in Spanish; er/sie/xier/sier in German). Each of five Llama-based models is prompted 50 times per condition, giving about 300,000 generations; generations are cleaned and clustered into 22 U.S. Bureau of Labor Statistics occupation categories, and bias is quantified by an $L^2$ norm and Jensen-Shannon divergence against a uniform 'ideal' distribution. The analysis repeatedly compares marginal (single-axis) distributions with the joint country-by-gender distribution, and this joint comparison is what surfaces biases hidden in single-axis checks.","core_discovery":"The paper claims to present the first multilingual study of intersecting country-gender bias in occupation recommendations, and its central discovery is that large language models exhibit intersectional country-gender occupational bias that is not reducible to gender bias or country bias alone. Using a benchmark of 300,000 prompts in English, Spanish, and German that vary 25 origin countries and four pronoun sets across five Llama-family models, the authors find that a model can show rough parity for gender or country taken separately while still channeling specific country-gender combinations into stereotyped occupations, such as low-status feminized jobs for women and non-binary people from some countries and high-status roles for men from Western countries. Instruction-tuned models (Alpaca and Llama3-Instruct) show the lowest and most stable bias levels, the prompt language changes the bias (Spanish least, English and German more), and host-country choice has at most a modulating effect. The paper concludes that single-axis, English-only fairness evaluations can miss compounded harms.","pith_inferences":["If fairness is instead defined as matching real-world labor-force distributions rather than a uniform spread, the paper's bias scores and model rankings could change substantially; testing that alternative baseline is a direct next step the authors leave open.","The Spanish-language result hints that models may fixate less on pronouns in pro-drop languages because pronouns are often omitted in natural text; prompting in a language without grammatical gender or without pronouns entirely would test this mechanism.","The same joint-distribution method could be applied to other job-relevant attributes, such as race or disability, where single-axis audits remain common, and could reveal analogous compounded harms.","Because the evaluation covers only Llama-family models, the instruction-tuning advantage may not generalize to other architecture and alignment families; comparing against differently trained models would show how general the mitigation is."],"forward_implications":["Fairness audits that report only gender parity or only country parity can certify models that still give stereotyped occupation advice to specific subgroups, so intersectional measurements should become standard in such audits.","Instruction tuning consistently lowered single-axis and intersectional bias in the tested suite, making tuning a concrete mitigation lever for occupation stereotyping even when it does not change model scale.","Bias measurements do not transfer across prompt languages: with the models that support all three languages, Spanish prompts showed the least bias, so a clean bill of health in English does not imply clean behavior in another language.","Host country had only a secondary, model-dependent effect, while prompt language and model family were the main drivers; this points to pretraining and prompting design as the active levers for bias rather than the user's location.","The released prompt framework, country set, and clustering pipeline can be reused to audit other models, languages, and occupation categories for the same intersectional effects."],"supporting_citations":[{"why":"Supplies the three English prompt templates and the occupation-recommendation framing that the paper translates and extends to more languages and pronouns.","marker":"Salinas et al. (2023)"},{"why":"Prior study of country and gender bias in classifiers using names as proxies; the paper contrasts its own no-names method against it.","marker":"Barriere and Cifuentes (2024)"},{"why":"Lays out challenges of bias evaluation across languages and cultures, motivating the multilingual design.","marker":"Talat et al. (2022)"},{"why":"Defines Llama2, the non-instruction-tuned English-heavy base model used as one comparison point.","marker":"Touvron et al. (2023)"},{"why":"Defines Alpaca, the instruction-tuned Llama2 variant used to isolate the effect of instruction tuning.","marker":"Taori et al. (2023)"},{"why":"Defines Latxa, a Llama2 model continually pretrained on Basque, used to test whether a genderless pretraining language reduces gendered output.","marker":"Etxaniz et al. (2024)"},{"why":"Defines Llama3, the multilingual base model evaluated in all three prompt languages.","marker":"Dubey et al. (2024)"},{"why":"Model card for Llama3-Instruct, the best-performing and least-biased model in the study.","marker":"AI@Meta (2024)"},{"why":"Provides the 22 occupation categories used for clustering and the uniform reference distribution that defines unbiased behavior.","marker":"U.S. Bureau of Labor Statistics (2024)"},{"why":"Provides the command-r-plus model used to clean responses and supervise the occupation clustering.","marker":"Cohere (2024)"}],"fun_headline_variants":["LLM job tips: hidden country-gender bias, even when axes look fair","Intersectional bias in LLM career advice slips past single-axis audits","First multilingual study: LLMs show compound bias in job recommendations","Country+gender biases in LLM job picks: instruction tuning reduces them","LLM occupation recs: intersectional bias persists in 3 languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole bias measurement rests on treating an equal spread of job categories across every country and gender group as the 'unbiased' ideal; if the right ideal is instead a distribution matching real labor markets or individual qualifications, the measured biases and model rankings could change.","fun_headline_variants_meta":{"raw":{"variants":["LLM job tips: hidden country-gender bias, even when axes look fair","Intersectional bias in LLM career advice slips past single-axis audits","First multilingual study: LLMs show compound bias in job recommendations","Country+gender biases in LLM job picks: instruction tuning reduces them","LLM occupation recs: intersectional bias persists in 3 languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000325,"raw_usage":{"total_tokens":1815,"prompt_tokens":929,"completion_tokens":886,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":790}},"tokens_in":545,"tokens_out":886,"duration_ms":9969,"temperature":1.0,"reasoning_tokens":790,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:50:08.740413+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 300,000-prompt benchmark but score bias against actual employment distributions by country and gender instead of a uniform ideal, and check whether the intersectional gaps persist and whether instruction-tuned models still look best; alternatively, on models that show single-axis parity, test conditional independence of occupation from the country-gender pair with a chi-square or permutation test, because if no dependence remains after accounting for the marginal gender and country distributions, the claim of hidden intersectional bias falls.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior study of country and gender bias in classifiers using names as proxies; the paper contrasts its own no-names method against it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Model card for Llama3-Instruct, the best-performing and least-biased model in the study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the command-r-plus model used to clean responses and supervise the occupation clustering."}],"review_version":1}