{"id":"97b5fdc3-c969-4751-95ad-74d461feae3e","arxiv_id":"2509.03329","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A culturally grounded Spanish benchmark of 4,156 prompts shows commercial LLMs exhibit different and often stronger bias in Spanish than in English, and that bias persists across sampling temperatures.","lead":"This paper introduces SESGO, a 4,156-prompt Spanish benchmark that adapts the BBQ methodology with Latin American sayings and stereotypes to test bias in six commercial LLMs. It reports that English-oriented bias mitigation does not transfer cleanly to Spanish and that temperature has little effect on measured bias.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated gold labels are the load-bearing assumption: every SESGO bias score inherits the dataset's Target/Other/Unknown labels, which lack expert or community validation.","rationale":"The reader's weakest assumption correctly identifies the unvalidated ground truth as the most load-bearing link in the paper's chain. The dataset is the paper's core contribution, and every empirical result—model rankings, category comparisons, cross-lingual transfer, temperature stability—is computed against the authors' gold labels. If those labels do not reflect genuinely held Latin American stereotypes, then the central claim collapses regardless of how carefully the models are prompted or how reasonable the bias metric looks. The authors' own limitation statement concedes that the prompts were not validated through user studies or expert reviews, which is an explicit missing-support flag. I also considered the separate concern that the 'bias mitigation techniques optimized for English fail to transfer' claim is not supported by any direct test of a mitigation technique; that is a real overstatement, but it is secondary because label validity is prior—if the labels are unreliable, that conclusion is unreliable too, and if the labels are reliable, the mitigation-transfer conclusion is at least an interpretable observation about deployed models. The paper does have independent support: the dataset and code are public, the framework is modular, and the metric is clearly specified, so the appropriate response is to condition acceptance on independent label validation rather than to reject outright.","tokens_in":18256,"tokens_out":7289,"duration_ms":69668,"concrete_test":"Select a stratified random sample of, say, 100 prompts per bias category (400 total). Have at least three independent annotators with Latin American backgrounds—diverse by country, region, and class—apply the paper's annotation rubric and produce gold labels blind to the authors' labels, and measure Fleiss' kappa separately for ambiguous and disambiguated prompts. If kappa is below about 0.7, or if a substantial fraction of ambiguous prompts receive a definite Target/Other label by annotators, the ground truth is not stable and the reported bias scores should be re-estimated on the subset with high agreement; large shifts would indicate the findings are label-driven.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the SESGO gold labels—Unknown for ambiguous contexts, Target/Other for disambiguated ones—are culturally valid encodings of Latin American stereotypes. All accuracy and bias scores (Tables 2–3, Figures 3–7) are computed relative to these labels; a systematic mislabeling of any class changes every comparison, including the cross-lingual and temperature claims. The paper reports no inter-annotator agreement, no pilot validation, and no expert review of the 4,156 prompts. The authors themselves note in the Discussion that the prompts 'could be refined through user studies or expert reviews ... to validate the representativeness of the dataset and bias categories.' The risk is not abstract: for ambiguous prompts, the correct answer is assumed to be Unknown whenever context is underspecified, but if a prompt embeds a culturally recognizable saying that makes one demographic answer inferable, forcing Unknown as gold would label a culturally competent response as an error and inflate both accuracy and bias scores for whichever model picks that answer. Disambiguated labels are equally sensitive: the added 'objective' context is written by the dataset authors, and there is no external check that the Target/Other assignments match how Latin American speakers would actually interpret the scenario. Since the paper's central claim depends on these labels, the dataset needs independent validation before the headline conclusions can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SESGO, a Spanish-language benchmark and evaluation framework for measuring stereotypical bias in instruction-tuned LLMs. The dataset contains 4,156 prompts built from Latin American sayings and cultural expressions, organized into four bias categories (racism, gender, classism, xenophobia), each with ambiguous and disambiguated versions and Target/Other/Unknown answer options. The authors define a bias score (Eq. 1) that combines accuracy with the difference between two error-direction metrics, evaluate six commercial or openly available models, and report (i) that bias is higher on ambiguous Spanish prompts, (ii) that matched Spanish/English comparisons reveal cross-lingual differences including direction reversals, and (iii) that bias scores are largely stable across sampling temperatures. The code and dataset are publicly released.","tokens_in":18492,"tokens_out":7957,"duration_ms":71361,"significance":"If the gold labels are culturally valid, SESGO is a valuable resource: it moves beyond translated English benchmarks, documents region-specific stereotypes, and offers a simple interpretable metric that separates accuracy from error direction. The matched English/Spanish comparison is a useful design for isolating language effects, and the public code and data release support reproducibility. However, the headline claims are not yet established: the correctness of every score depends on unvalidated labels, the cross-lingual and temperature comparisons lack uncertainty quantification, and the claimed failure of English-optimized mitigation is not directly tested. With label validation and more careful statistical reporting, the contribution would be solid.","major_comments":[{"comment":"All accuracy and bias scores in Tables 2-3 and Figures 3-7 inherit the gold labels Target/Other/Unknown, but no validation of these labels is reported: there is no inter-annotator agreement, expert review, or pilot study, and the annotation protocol is not described. The authors themselves state in the Discussion that the prompts 'could be refined through user studies or expert reviews ... to validate the representativeness of the dataset and bias categories.' This is load-bearing rather than a routine limitation: if a nontrivial fraction of ambiguous prompts actually contains a culturally recognizable stereotype that makes a demographic answer inferable, then treating Unknown as the gold answer would penalize culturally competent responses and inflate bias scores for models that choose that answer. Please add validation of the labels, or at minimum report agreement and a sensitivity analysis showing that the main cross-lingual and temperature conclusions are robust to plausible label noise.","section":"SESGO Dataset and Discussion (limitations)"},{"comment":"The conclusion and abstract claim that 'bias mitigation techniques optimized for English fail to transfer effectively to Spanish contexts,' but the paper does not apply or ablate any mitigation technique. Figures 5-6 and Tables A1-A2 compare model behavior on English versus Spanish matched prompts; this is an observational language comparison, not a test of mitigation transfer. The claim should be reframed as cross-lingual bias differences, or the experiment should actually include an English-optimized mitigation intervention and measure its effect on the Spanish prompts.","section":"Cross-Linguistic Transferability of Bias Mitigation"},{"comment":"The claim that bias scores 'remain remarkably stable across temperature values from 0.1 to 1' is not supported by the reported point estimates. In Table A3, DeepSeek R1's disambiguated bias score moves from -0.243 (T=0.1) to +0.261 (T=0.5) and 0.000 (T=1.0), including sign reversals; other models also show small but non-negligible variation. Moreover, no confidence intervals, standard errors, or significance tests are reported for any temperature comparison, and the models are stochastic, so repeated sampling is needed. Please add uncertainty quantification and soften or revise the stability claim accordingly.","section":"Bias Presence Across Sampling Temperatures, Table A3"},{"comment":"Equation (1) is not fully specified. The text describes the ideal model as the point (acc=1, F(Target)=F(Other)), but that condition is a line, not a point; the formula sqrt((1-acc)^2 + (F(Target)-F(Other))^2) is not the Euclidean distance to (1,0,0) in the (F(Target), F(Other)) plane, and the sign sigma is introduced but never defined in terms of the response data. Since every reported bias score in Tables 2-3 and Figures 3-7 uses this metric, please provide an explicit definition of sigma (e.g., sign(F(Target)-F(Other)) with a stated convention) and explain the intended geometry, including how the equality F(Target)+F(Other)=1-acc is used.","section":"Metrics for Bias Quantification, Eq. (1)"}],"minor_comments":[{"comment":"There are typos in the text: 'an LLMs risk perpetuating these biases' should be 'LLMs risk perpetuating these biases,' and 'whit indigenous and black communities' should be 'with indigenous and black communities.'","section":"Xenophobia and Racism subsections"},{"comment":"The caption 'Best value for each bias category (in absolute value) in bold' is ambiguous because lower bias score is better but the tables also report direction; please state precisely what is bolded and why a nonzero signed score can be 'best.'","section":"Tables 2 and 3"},{"comment":"The appendix table reports a column labeled 'Ft-Fo' without defining it in the caption; please define F(Target), F(Other), and Ft-Fo in the appendix or refer explicitly to the main-text definitions.","section":"Appendix A3"},{"comment":"The Discussion states that 'bias scores consistently increase when models operate in ambiguous contexts in Spanish compared to English,' but the Cross-Linguistic section reports that three out of six models show this pattern in ambiguous contexts; please correct this inconsistency so the Discussion matches the evidence.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The dataset and metric are potentially valuable, but the contribution should be positioned more carefully against MBBQ, which already includes Spanish. The main obstacles are the unvalidated gold labels and the absence of uncertainty quantification for the cross-lingual and temperature claims. I would support publication after these points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nSESGO is worth reading if you do multilingual bias eval. The contribution is the dataset: more than 4,000 Spanish prompts built around documented Latin American sayings and stereotypes, with ambiguous/disambiguated pairs, plus a chunk of BBQ-derived matched items for cross-linguistic comparison. The error-direction metric (bias score as Euclidean distance combining accuracy and F(Target)-F(Other) with a sign) is simple but practical, and the triangular constraint region in Figure 2 is a nice touch. They also ship the code and data, which makes the empirical claims checkable.\n\nWhat's genuinely new: culturally authored prompts rather than direct translation, and the emphasis on xenophobia toward Venezuelan migrants, which BBQ-type benchmarks can't capture. The finding that bias patterns hold across temperature (0.1–1.0) is consistent with prior work, and their matched English/Spanish comparison is a reasonable design.\n\nThe soft spots are mostly about the strength of the claims. The abstract says bias mitigation \"fails to transfer\" but no mitigation intervention is tested; what they observe is that models show more bias on Spanish prompts than English prompts. That's evidence for a language gap, not for failure of a mitigation technique. They should either soften the language or run an actual intervention. The same section lacks confidence intervals or significance tests; with six models and category-level scores, some differences may be noise. Fixable, but should be required.\n\nThe bigger issue is the gold labels. Every accuracy and bias score is computed against Target/Other/Unknown labels assigned by the authors. For ambiguous prompts, the gold answer is always Unknown. If some prompts embed a saying that makes a demographic attribution inferable to actual Spanish speakers, then a culturally competent model would be marked wrong for giving that answer. The authors acknowledge in the Discussion that the dataset hasn't been validated through user studies or expert review. That is the right caveat, but it should move to the front and be treated as a condition for the headline results, not an afterthought. The xenophobia prompts also come from a proprietary social-media dataset (El Barómetro) with no public access, so independent checking of those 1,344 items is limited.\n\nProportionately: this is not a broken paper. The central mechanism—culturally grounded prompts reveal patterns translated benchmarks miss—is plausible and the results are internally consistent. The fixes are validation of a sample of labels, significance tests, and reworded claims.\n\nWho should read it: anyone building multilingual fairness benchmarks, and people auditing Spanish- or Latin American-deployed systems. I'd bring it to a reading group and cite it if we work in that area. For peer review: yes, a serious editor should send it to referees. The dataset and metric deserve scrutiny and likely publication after revision.","headline":"A useful, reusable Spanish bias benchmark with a sensible metric; the dataset labels carry the weight, and the cross-lingual 'mitigation' claim outruns the evidence.","tokens_in":18983,"tokens_out":1736,"would_cite":true,"duration_ms":15645,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 4,156-prompt Spanish bias evaluation finds that culturally specific Latin American stereotypes expose bias that translated English benchmarks miss.","keywords":["Spanish language","social bias","large language models","BBQ benchmark","Latin American stereotypes","cross-lingual transfer","bias evaluation metric","xenophobia"],"falsifier":"Take a random subset of SESGO prompts and have independent raters from several Latin American countries assign Target, Other, Unknown, and the correct disambiguated answer without seeing model results, then recompute all bias scores with those labels. If the direction or magnitude of model rankings changes substantially under alternative ground truth, the reported bias patterns depend on the authors' stereotype readings rather than on stable model behavior.","tokens_in":18087,"feed_emoji":"⚖️","tokens_out":4848,"duration_ms":44552,"temperature":0.7,"pith_summary":"The paper claims that current bias evaluations are too English-centric and that commercial large language models show different, often stronger, bias patterns when prompted in Spanish with culturally specific Latin American stereotypes. It introduces SESGO, a dataset of 4,156 prompts built from regional sayings and expressions, and a bias metric that combines accuracy with the direction of errors. Testing six instruction-tuned models, it reports that bias scores are higher on culturally grounded Spanish prompts than on translated English ones, that English-optimized mitigation does not transfer reliably, and that sampling temperature has little effect on bias. If correct, Spanish-speaking users face harms that English benchmarks systematically miss.","feed_headline":"Spanish prompts expose bias English benchmarks miss","feed_subtitle":"A 4,156-prompt study of six chatbots shows English-tuned safety fails to transfer to Latin American Spanish.","key_machinery":"The central machinery is the SESGO dataset together with a signed bias score. SESGO builds scenarios from documented Latin American sayings, such as “A Black man without a master is like a child without a father,” across gender, race, class, and xenophobia, and each prompt appears in ambiguous and disambiguated forms with positive and negative questions and Target, Other, and Unknown answer options. The bias score is the Euclidean distance between the ideal point, where accuracy is 1 and $F(\\mathrm{Target}) = F(\\mathrm{Other})$, and the observed point, with a sign indicating which group bears the detected bias: bias score $= \\sigma \\sqrt{(1-\\mathrm{acc})^{2} + (F(\\mathrm{Target})-F(\\mathrm{Other}))^{2}}$. This metric is what lets the paper separate raw accuracy from the direction of model errors.","core_discovery":"On the paper's own terms, the discovery is that culturally situated Spanish prompts reveal bias patterns that translated benchmarks obscure. Using ambiguous and disambiguated questions adapted from the BBQ methodology but grounded in Latin American popular sayings, the authors find that several leading models answer incorrectly more often in Spanish and, when they err, target historically marginalized groups more strongly. They also find that bias scores rise on full culturally grounded prompts compared with direct translations, and that temperature changes do not materially shift bias. The paper presents this as the first systematic evaluation of commercial LLMs on culturally specific bias in Spanish.","pith_inferences":["Beyond the paper: if the dataset's labels encode Colombian-centric readings of sayings, bias scores could shift when the same items are rated by people in Mexico, Argentina, or Spain; a multi-country label validation would test how much of the reported effect is regional.","Beyond the paper: the findings predict that harmful chatbot outputs in Spanish will concentrate in narratives tied to local out-groups, such as Venezuelan migrants in Colombia, rather than in generic translated stereotypes.","Beyond the paper: the same accuracy-plus-error-direction metric could be applied to culturally grounded prompt sets in other regional or Indigenous languages to map where English-centric mitigation fails most."],"forward_implications":["A model that appears balanced on translated English prompts can still show systematic bias on culturally grounded Spanish prompts.","English-language safety training cannot be assumed to protect Spanish-speaking users; mitigation must be validated on native-language, culturally specific data.","Because temperature does not change bias scores, decoding-time adjustments are not a practical lever; the levers are upstream training and fine-tuning.","The modular prompt structure can be extended to new stereotypes, bias categories, and languages, giving other regions a ready-made template for culturally aware evaluation.","Benchmarks that claim multilingual coverage by translation should be read as measuring linguistic equivalence, not cultural safety."],"supporting_citations":[{"why":"Supplies the underspecified-question methodology and the Target/Other/Unknown answer scheme that SESGO adapts and extends.","marker":"Parrish et al. 2022"},{"why":"Documents that GPT-4 safety mitigations were designed, built, and tested primarily in English with a US-centric view, motivating the cross-linguistic test.","marker":"Achiam et al. 2023"},{"why":"Provides MBBQ, a multilingual BBQ translation including Spanish, which serves as the translation-based baseline the paper contrasts with culturally grounded prompts.","marker":"Neplenbroek, Bisazza, and Fernández 2024"},{"why":"Surveys bias and fairness in large language models, supporting the need for generation-based, culturally specific evaluation.","marker":"Gallegos et al. 2024"},{"why":"Gives prior evidence of persistent gender bias in Spanish language models, reinforcing the case for Spanish-specific bias testing.","marker":"Garrido-Muñoz, Martínez-Santiago, and Montejo-Ráez 2024"},{"why":"Documents racist sayings and stereotypes in Latin American discourse that SESGO uses to construct its racism prompts.","marker":"Castellanos Guerrero and Landázury Benítez 2012"},{"why":"Provides documented xenophobic narratives from Latin American social media that SESGO uses as templates for its xenophobia prompts.","marker":"El Barómetro 2024"}],"fun_headline_variants":["Spanish sayings expose LLM bias English tests miss","Bias in Spanish chatbots: English mitigation fails","First Spanish cultural bias probe of big LLMs","Temperature won't fix bias in Spanish LLMs","Culturally Spanish prompts reveal hidden chatbot bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every SESGO ground-truth label—who is the target of a saying, what the correct disambiguated answer is, and when Unknown is right—accurately encodes a genuinely held Latin American stereotype; the authors note these labels were not validated through user studies or expert review.","fun_headline_variants_meta":{"raw":{"variants":["Spanish sayings expose LLM bias English tests miss","Bias in Spanish chatbots: English mitigation fails","First Spanish cultural bias probe of big LLMs","Temperature won't fix bias in Spanish LLMs","Culturally Spanish prompts reveal hidden chatbot bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1270,"prompt_tokens":900,"completion_tokens":370,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":298}},"tokens_in":516,"tokens_out":370,"duration_ms":3665,"temperature":1.0,"reasoning_tokens":298,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:29:57.304675+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random subset of SESGO prompts and have independent raters from several Latin American countries assign Target, Other, Unknown, and the correct disambiguated answer without seeing model results, then recompute all bias scores with those labels. If the direction or magnitude of model rankings changes substantially under alternative ground truth, the reported bias patterns depend on the authors' stereotype readings rather than on stable model behavior.","supporting_citations":[{"cited_title":"M.; and Bowman, S","cited_arxiv_id":null,"evidence_quote":"Supplies the underspecified-question methodology and the Target/Other/Unknown answer scheme that SESGO adapts and extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides MBBQ, a multilingual BBQ translation including Spanish, which serves as the translation-based baseline the paper contrasts with culturally grounded prompts."},{"cited_title":"O.; Rossi, R","cited_arxiv_id":null,"evidence_quote":"Surveys bias and fairness in large language models, supporting the need for generation-based, culturally specific evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives prior evidence of persistent gender bias in Spanish language models, reinforcing the case for Spanish-specific bias testing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents racist sayings and stereotypes in Latin American discourse that SESGO uses to construct its racism prompts."}],"review_version":2}