{"id":"d0e754e0-f345-4037-abe8-1c42b10504a0","arxiv_id":"2412.08846","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Open-source LLMs match human cultural judgments better for Western countries and socio-cultural norms, and multilingual training, synthetic data, scale, and alignment each appear to improve cultural alignment.","lead":"This paper compares how open-source language models (LLaMA, Yi, Phi) answer World Values Survey questions against human respondents in 55 countries, then relates the match quality to model size, training data, and alignment. It finds models are close to humans on socio-cultural norms, biased toward Western views, and that multilingual or synthetic data can improve cultural alignment.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"English-only probing confounds language with cultural values; the Western-bias and category-similarity claims need local-language WVS prompts before they can be accepted.","rationale":"The reader's weakest_assumption is that ordinal WVS answers can be converted to interval scores and that token likelihoods reflect the model's actual judgment. I partially agree: that is a real concern, but the more specific and more consequential unexamined factor is the English-only prompting template. Even if the equal-spaced scores were accepted, correlations with human responses collected in many languages would still be difficult to interpret because the model is always probed in English. The paper's own Obs. 3 depends on the observation that multilingual training improves non-Western correlations under English prompts, which suggests the probe language is not neutral; that makes local-language probing a necessary control before attributing the effect to training corpus or model size. This is a correctness risk rather than an internal inconsistency, since the authors are transparent about the template, but it directly threatens the strongest_claim's Western-bias assertion and the category-level similarity ranking. I would keep the reader's CONDITIONAL verdict: the observations are worth reporting as exploratory, but the headline findings should not be stated as established until local-language probing and scoring robustness are reported. No additional verdict change is needed beyond what the reader already recommended.","tokens_in":6679,"tokens_out":6648,"duration_ms":81422,"concrete_test":"Using the official WVS translations, probe Llama-3-70B-Chat, Llama-2-70B-Chat, and Yi-34B-Chat on a fixed subset of about 50 WVS items in 6-8 countries spanning Western and non-Western regions (e.g., USA, Germany, Mexico, Egypt, Kenya, South Korea, Japan, China), in both English and each country's WVS survey language, with the same multiple-choice template. Recompute the per-country Pearson correlations of Figs. 2-3. If the Western-vs-non-Western gap changes materially (e.g., by more than 0.1 in Pearson r) or the category ordering in Fig. 1 changes, the English-only probe is a first-order confound; if the pattern is unchanged, the Western-bias claim is robust to language of elicitation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All country-level judgments are elicited with one English template (Appendix A.1, Fig. 5), while the human benchmark comes from WVS interviews conducted in respondents' own languages. The central claims—especially Obs. 2 and Obs. 3—therefore conflate the models' cultural values with the language of elicitation. A model asked \"In South Korea, Answer:\" in English may retrieve English-text associations about Korea, or respond to the English wording of the answer options, inflating correlations with English-speaking/Western countries and deflating them elsewhere. The same issue touches Obs. 1: categories judged \"more similar\" may simply be those whose English wording is most directly translatable, not necessarily those the model understands better. The scoring scheme in Sec. 2.2 adds a second unvalidated layer: ordinal answer labels are assigned equal-interval scores in [-1,1], and Eq. (1) uses raw next-token probability products without length or tokenization normalization; both can change the Pearson correlations in Figs. 1-4. The Limitations section only restricts the analysis to the WVS dataset and calls the synthetic-data analysis preliminary; it does not list prompt language or scoring robustness as caveats. Because every headline conclusion is a comparison of these correlations, an artifact in the English probe would propagate to all six observations.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper probes open-weight LLMs (Phi, Yi, Llama-2, Llama-3) on the World Values Survey by converting each question into a multiple-choice task with an English prompt template, assigning equally spaced scores in [-1,1] to ordinal answer choices, and computing LLM answer probabilities from next-token likelihoods. It then compares LLM and human mean scores via Pearson correlations. Six observations are reported: LLMs are more human-like on socio-cultural norms than on social systems and progress, their judgments are Western-biased, multilingual pretraining improves cultural diversity, larger models are more culturally aware, synthetic data can help smaller models, and alignment increases human-likeness. The paper acknowledges that the analysis is confined to WVS and that the synthetic-data analysis is preliminary.","tokens_in":6911,"tokens_out":3774,"duration_ms":43362,"significance":"If the measurement methodology is valid, the paper offers a practical and inexpensive way to measure cultural alignment of open LLMs and connects it to concrete training choices, which would be valuable for deploying LLMs across cultures. The use of multiple open model families with publicly known training configurations is a strength, and the WVS-based probing is transparent. However, the current evidence is suggestive rather than confirmatory: the core comparisons lack uncertainty quantification, the scoring model rests on unvalidated assumptions, and the English-only elicitation confounds language with cultural knowledge. With additional robustness checks and significance testing, the contribution could become a useful benchmark for cultural alignment in LLMs.","major_comments":[{"comment":"The paper assigns equally spaced scores in [-1,1] to ordinal WVS answer categories and defines pk as the raw product of next-token probabilities over the answer candidate before normalizing across candidates. Neither step is validated. If the true latent scale is not equally spaced, the Pearson correlations can change under alternative monotone spacings, and candidates with different token lengths receive incomparable products. Please provide robustness checks with alternative scoring schemes (e.g., rank-based or proportional-odds spacings) and length-normalized likelihoods, or explicitly justify the current choices with calibration evidence.","section":"Section 2.2, Eq. (1)"},{"comment":"All LLM judgments are elicited with a single English prompt template, 'In {Country}, Answer:', while the WVS human benchmark is collected through interviews in respondents' own languages. Obs. 2 and Obs. 3, and the category-level claim in Obs. 1, may therefore reflect the model's English-language associations with a country name rather than its cultural value judgments. Controlled local-language prompting on a subset of questions, or at least a clear statement of this confound in the Limitations section, is necessary before the cross-cultural claims can be accepted.","section":"Appendix A.1, Fig. 5"},{"comment":"The division of the 12 WVS categories into the 'Socio-Cultural Norms' group and the 'Social Systems and Progress' group is made after inspecting the same correlation results in Fig. 1, with no independent criterion or statistical test. This post hoc binning is not evidence for the categorical claim. The groups should be defined from the WVS category descriptions a priori, or the difference between them should be tested with an appropriate significance test that accounts for the data-driven choice.","section":"Section 3.2, Obs. 1 and Fig. 1"},{"comment":"Statements such as 'Yi models show significantly higher correlations' (Obs. 3), 'Yi-34B-Chat achieves a significantly higher correlation than Yi-6B-Chat' (Obs. 4), and 'both models show much higher correlations when fine-tuned as chat models' (Obs. 6) are unsupported because the reported figures show point estimates without confidence intervals, standard errors, or significance tests. The paper should report uncertainty and perform hypothesis tests, accounting for the number of comparisons across models, continents, and categories.","section":"Figures 2-4 and Obs. 3-6"},{"comment":"The comparison between phi-3-3.8B (trained on synthetic data) and Llama-3-8B (trained on multilingual data) confounds training-data composition with model family, tokenizer, training compute, and many other design choices. The paper itself notes that the synthetic-data details are undisclosed and that the Phi provenance is a guess. The claim that synthetic data can help smaller models overcome size limits would require controlled comparisons, for example by training the same architecture on otherwise matched synthetic versus web data, or by clearly presenting the result as a case-study hypothesis rather than an observation.","section":"Obs. 5 and Table 1"}],"minor_comments":[{"comment":"There is a typo in 'we regularly distributes1, . . . , sK over interval [−1, 1]'; 'distributes' should be 'distribute', and the sentence should be reworded for clarity.","section":"Section 2.2"},{"comment":"The caption says 'The Bold with underline denotes the group of large models,' but the table formatting is not reproduced in the text; please ensure the intended bold/underline is visible or describe the grouping explicitly.","section":"Appendix A.2, Table 1"},{"comment":"The phrase 'Postmaterialist indes' should be 'Postmaterialist index'.","section":"Appendix B.2, Table 4"},{"comment":"The reference for 'Y Liu. 2019. Roberta: A robustly optimized bert pretraining approach' omits the full author list and would be clearer with the standard arXiv citation.","section":"References"},{"comment":"The dataset name is 'World Values Survey' (plural), but the paper consistently writes 'World Value Survey'; please correct the name throughout.","section":"Introduction and Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important question, and the honest style of the observations is appreciated. However, the English-only prompt is a serious confound for all cross-cultural claims, and the lack of uncertainty quantification weakens the headline observations. In a revision, I would expect at least a focused multilingual prompting experiment and confidence intervals or significance tests for the main comparisons; if those cannot be supplied, the paper should be substantially reframed as a preliminary exploratory study rather than a set of findings."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a clearly written empirical survey of how recent open LLMs judge WVS cultural-value items, and it tries to connect those judgments to training choices (multilingual data, size, synthetic data, alignment). The headline claims—LLMs are closer to humans on socio-cultural norms than on social systems/progress, they are Western-biased, and certain training choices help—are plausible but not established by the evidence as presented. The biggest unaddressed problem is that every country-level judgment is elicited with a single English template (Fig. 5, Appendix A.1), while the human benchmark comes from WVS interviews in respondents' own languages. That confounds language with cultural values. A model asked about South Korea in English may recall English-language text about Korea, or react to the English wording of answer options, which could inflate correlations with Western countries and deflate them elsewhere. This directly threatens Obs. 2 and Obs. 3, and also Obs. 1, because categories that look 'more similar' may simply be those whose English wording translates most directly.\n\nWhat's genuinely new: applying this probing approach to Llama-3, Phi-3, and Yi, and reading training methodology into the results. Prior work on BERT/RoBERTa documented Western bias and limited human-likeness but didn't link to training choices. The observation that Yi, trained on a large Chinese corpus, correlates better with non-Western cultures is suggestive and worth following up. The paper is honest about being limited to WVS and about the synthetic-data analysis being preliminary.\n\nThe soft spots are more than cosmetic. There are no error bars, confidence intervals, or significance tests anywhere in Figs. 2-4, so 'significantly higher' is unsupported. Obs. 1 bins categories after seeing the data. The scoring scheme assumes equal spacing of ordinal WVS answers in [-1,1] and uses raw next-token probability products without length normalization; a robustness check is needed. The training-methodology claims compare different model families with different data mixes, so model size and multilingual data are confounded. And no code or data is released.\n\nNone of this kills the paper. The directional claims are consistent with earlier findings, and the questions asked are the right ones. But as it stands this reads as exploratory observation, not measured result. With local-language probes, significance testing, and robustness checks, it could become a solid contribution.\n\nWorth sending to peer review rather than desk rejecting, because the question matters and the paper has real content. I'd send it out expecting heavy revision. I'd bring it to reading group as a cautionary example of probing methodology, not because the results are decisive. I wouldn't cite it yet.","headline":"Clear empirical survey of LLM cultural alignment on WVS, but the English-only probe and missing statistical tests make the headline conclusions provisional.","tokens_in":7445,"tokens_out":2756,"would_cite":false,"duration_ms":27825,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims open LLMs judge cultural values like humans, diverge on politics and science, skew Western, and can be improved by multilingual training, scale, synthetic data, and alignment.","keywords":["large language models","cultural values","World Values Survey","cross-cultural bias","multilingual training","synthetic data","alignment","model scaling"],"falsifier":"Re-run the identical probe with each question's answer labels permuted across the K choices; a model with genuine cultural values should give the same human correlation regardless of label order, whereas a token-likelihood artifact would not. As a companion check, compute Spearman rank correlations on the same data with no equal-spacing assumption and see whether the ranking of models and categories survives.","tokens_in":6455,"feed_emoji":"🌍","tokens_out":10442,"duration_ms":94439,"temperature":0.7,"pith_summary":"This paper asks whether open-source LLMs hold human-like cultural values and whether training choices can change those values. Using the World Values Survey's 209 questions across 55 countries, the authors convert each question into a multiple-choice probe and compare models' mean answers with human survey responses. They report that LLM judgments track humans on socio-cultural norms such as ethics and security, but diverge on political systems, economics, and science; that judgments skew toward Western countries; and that multilingual pretraining, larger scale, synthetic data, and alignment each improve human-likeness. If these findings hold, cultural alignment is not a fixed property of a model but a measurable and steerable design objective for open LLMs.","feed_headline":"LLMs mirror human culture but skew Western; multilingual data helps","feed_subtitle":"Bigger models, alignment, and multilingual data each move those judgments closer to human answers.","key_machinery":"The measuring instrument is a WVS multiple-choice probe. Each survey question is turned into a template prompt ending with \"In {Country}, Answer:\" and each candidate answer is assigned an equally spaced score in the interval [-1, 1], such as never=-1, neutral=0, and justifiable=1. The model's probability of choosing answer k is the normalized product of next-token likelihoods over the answer string (Eq. 1), giving each question a model mean score, while humans get a mean score from the survey's empirical answer distribution. Pearson correlations between the two sets of mean scores, grouped by WVS category and by continent, produce the similarity rankings that all six observations rest on. The model families Llama-2, Llama-3, Yi, and Phi act as natural experiments because their public training-token counts, multilingual composition, sizes, and chat versus non-chat versions vary independently.","core_discovery":"The paper's central discovery is that cultural values in open-source LLMs are both measurable and trainable. Across 209 World Values Survey questions in 55 countries, the authors report six regularities: LLM mean scores correlate strongly with human mean scores on socio-cultural norm categories like ethical values and security, but weakly on politics, economics, and science; correlations are higher for Western countries than for non-Western, especially Asian, countries; models trained on large multilingual corpora, such as Yi's English plus Chinese data or Llama-3's 30-plus languages, show improved non-Western correlations, suggesting cross-cultural knowledge transfers across languages; larger models correlate more with humans than smaller ones; a 3.8B model trained largely on synthetic data can beat larger web-trained models on human-likeness, so scale limits can be overcome; and chat-aligned versions of the same base models match human judgments much better than their unaligned counterparts. Together these claims state that an open LLM's cultural alignment is a design parameter, not a fixed accident of pretraining.","pith_inferences":["The equal-spacing scoring assumption is untested: re-scoring the same answers with rank-based Spearman correlations, or with question-specific non-uniform weights, would show whether the observed model rankings are artifacts of the [-1, 1] spacing.","Because the probe always names a country in the prompt, the measured values are the model's country-conditional judgments; it remains open whether unprompted default answers would show more or less Western bias.","The paper's own limitations section notes that only the WVS is used and that Phi's synthetic-data recipe is undisclosed, so the causal claims about synthetic data are preliminary; repeating the probe on a second cultural-values survey and on a synthetic-data corpus with known generation settings would test them directly.","If cultural alignment is trainable, cultural-value benchmarks could usefully sit alongside reasoning benchmarks when choosing training corpora; otherwise gains in reasoning may hide divergence in cultural values."],"forward_implications":["A designer can evaluate an open LLM's cultural alignment with this probe before deployment, using existing survey data rather than collecting new human judgments.","Adding substantial non-English data to pretraining should shift model judgments toward non-Western human responses, even when the prompts are in English.","Larger parameter counts are not the only route to cultural human-likeness: synthetic data distilled from larger models can bring small models close to or above larger web-trained models.","Alignment through chat fine-tuning moves model judgments closer to human survey responses across cultural categories.","Questions about political regimes and science and technology remain difficult even for large, aligned models, so those areas are the most likely places for cultural misalignment to surface in real applications."],"supporting_citations":[{"why":"Supplies the 209 WVS questions, 55 countries, and human answer distributions that every model-versus-human correlation is computed against.","marker":"Haerpfer et al., 2022"},{"why":"Provides the Pearson correlation formula used to quantify agreement between model mean scores and human mean scores.","marker":"Freedman et al., 2007"},{"why":"Defines the Llama-2 family and its pretraining corpus; the paper's size and chat versus non-chat comparisons rest on these models.","marker":"Touvron et al., 2023"},{"why":"Defines Llama-3 and its multilingual corpus of over 30 languages, supporting the claim that multilingual data improves non-Western correlation.","marker":"Dubey et al., 2024"},{"why":"Defines the Yi models and documents their English-plus-Chinese training corpus, the main evidence for multilingual boosting of non-Western values.","marker":"Young et al., 2024"},{"why":"Defines Phi-3 and its synthetic-data-heavy training, the central evidence that small models can beat larger web-trained models.","marker":"Abdin et al., 2024"},{"why":"Documents Phi-1.5's textbook-style data, used in the small-model comparisons of the synthetic-data observation.","marker":"Li et al., 2023"},{"why":"Documents Phi-2's synthetic textbook data, used in the same small-model comparison.","marker":"Gunasekar et al., 2023"},{"why":"Provides the paper's indirect rationale for assuming Phi's synthetic data is generated by larger models.","marker":"Ben Allal et al., 2024"},{"why":"Supports the claim that multilingual training transfers cross-cultural knowledge across languages.","marker":"Zhao et al., 2024"}],"fun_headline_variants":["Multilingual training corrects LLMs' Western cultural bias","LLM cultural bias shrinks with multilingual and bigger models","Study: LLM cultural alignment improves with training data","Making LLMs less Western: multilingual and synthetic data help"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the survey's ordinal answer choices can be scored as equally spaced numbers from -1 to 1 and that the model's normalized likelihood of generating each answer phrase is its probability of choosing that answer; if either fails, the Pearson correlations do not measure cultural human-likeness as claimed.","fun_headline_variants_meta":{"raw":{"variants":["Multilingual training corrects LLMs' Western cultural bias","LLM cultural bias shrinks with multilingual and bigger models","Study: LLM cultural alignment improves with training data","Making LLMs less Western: multilingual and synthetic data help"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000257,"raw_usage":{"total_tokens":1549,"prompt_tokens":888,"completion_tokens":661,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":595}},"tokens_in":504,"tokens_out":661,"duration_ms":7520,"temperature":1.0,"reasoning_tokens":595,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:28:37.152037+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the identical probe with each question's answer labels permuted across the K choices; a model with genuine cultural values should give the same human correlation regardless of label order, whereas a token-likelihood artifact would not. As a companion check, compute Spearman rank correlations on the same data with no equal-spacing assumption and see whether the ranking of models and categories survives.","supporting_citations":[],"review_version":1}