{"id":"68aee2c9-119e-4dbc-b91a-a10a02061036","arxiv_id":"2412.00956","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Current open language models, especially English-centric GPT-2 and OPT, show weak or negative correlations with cross-national moral survey values, and the best model, BLOOMZ-560M, reaches only small positive correlations.","lead":"Researchers tested four language model families against moral surveys from over 40 countries and found that the models mostly fail to mirror national moral stances, with the multilingual BLOOMZ model doing best but still weakly. The study confirms that current open language models carry a positivity bias and do not capture fine-grained cross-cultural moral variation.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 4.2 defines the moral score as log P(moral token) minus log P(nonmoral token), but for multiword pairs 1, 3, and 4 the scored token is ambiguous; with no baseline control for token frequency or positivity bias, the averaged correlations do not establish that LLMs capture cultural moral…","rationale":"The reader's weakest_assumption correctly identifies moral-score validity as the load-bearing point. I agree with that reading. The additional specificity here is that Section 4.2's scoring is not merely noisy but formally ambiguous for three of the five token pairs, and the absence of any baseline control means the reported correlations could be entirely explained by English token priors or a positivity bias. The WVS missing-value recoding in Section 4.1 (replacing 'Don't know' and 'Not asked in survey' with zero) is a further ground-truth problem, but the score-invalidity issue is more fundamental because it affects every reported correlation. Since the paper does not release code or data and does not specify how multiword phrases are scored, the central claim cannot be verified from the manuscript as written; hence I would move from CONDITIONAL to UNVERDICTED rather than simply adding conditions. If the authors supply the exact scoring procedure and the proposed baseline controls, the verdict could return to CONDITIONAL or ACCEPT depending on the results.","tokens_in":13837,"tokens_out":9561,"duration_ms":92308,"concrete_test":"Recompute the moral score for a single fixed prompt, e.g. 'In Germany, getting a divorce is', and report the raw log-probabilities used for each member of pair3 and pair4. If the scored value is the first-token probability, the pair3 and pair4 differences are zero, which would invalidate the averaged correlations. Then recompute all reported correlations using only the single-token pairs (right, wrong) and (ethical, unethical), and run a control condition in which the country name is replaced by a random country or omitted. If the BLOOMZ averaged correlations drop below roughly 0.1 or lose significance under either check, the claimed cultural alignment is an artifact of token priors or topic-level positivity bias rather than country-specific moral knowledge.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central quantitative claim rests entirely on the moral_score measure defined in Section 4.2, computed as the average over five token pairs of log P(moral token) minus log P(nonmoral token). For pair1 ('always justifiable' vs 'never justifiable'), pair3 ('morally good' vs 'morally bad'), and pair4 ('ethically right' vs 'ethically wrong'), these are not single tokens. The paper says 'the logit corresponding to the word appearing after the prompt is calculated,' which is ambiguous. If only the first token is scored, pair3 and pair4 have identical first tokens ('morally'/'ethically') in both alternatives, making the difference identically zero and the reported nonzero correlations impossible; if the full phrase is scored as a sequence, the required computation is not described. Moreover, no baseline is subtracted: the difference log P(moral) minus log P(nonmoral) for a single-token pair like (right, wrong) can be dominated by the marginal frequency of the words in English, independent of country or topic. Correlating this unadjusted score with survey means across country-topic pairs can therefore yield spurious alignment driven by topic-level priors or a general positivity bias. The conclusion that BLOOMZ 'shows the best performance' with averaged correlations of 0.25/0.29 on WVS is uninterpretable until the token scoring is precisely specified and a country-agnostic baseline is shown not to reproduce the correlations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper probes four autoregressive language models (GPT-2 variants, OPT variants, Qwen2-0.5B, BLOOMZ-560M) for cross-cultural moral knowledge by scoring English prompts of the form 'In {country} {topic} is {moral_judgment}' and 'People in {country} believe {topic} is {moral_judgment}.', computing a 'moral score' as the difference in log-probabilities between morally charged and non-morally charged token pairs, and correlating these scores with country-topic means from the World Values Survey Wave 7 and the PEW 2013 Global Attitudes Survey. The main finding is that most models show weak or negative correlations with survey data, while BLOOMZ-560M shows the strongest positive correlations (e.g., 0.25 and 0.29 on WVS averaged token pairs), yet still falls short of moderate alignment. The paper concludes that current PLMs capture some but not reliable cross-cultural moral variation and that token-pair choice matters more than prompt phrasing.","tokens_in":14174,"tokens_out":3127,"duration_ms":31965,"significance":"If the measurement were validated, the paper would provide a useful replication and extension of prior work on cultural moral norms in LLMs, with an explicit comparison of monolingual and multilingual models across two large survey datasets. The study is transparent about model sizes and uses publicly available benchmarks, and the conclusion that even the best model only weakly aligns with survey ground truth is a sobering and potentially valuable result for the growing literature on cultural AI alignment. However, the paper's central quantitative claim rests on a moral-score computation that is ambiguously specified for multi-token pairs and that does not control for token-frequency or positivity baselines; without resolving these issues, the reported correlations (including the headline BLOOMZ result) are not interpretable as evidence about cultural moral knowledge. The paper also makes several non-defended preprocessing choices (replacing WVS missing values with 0, coding PEW non-responses as morally unacceptable) that can systematically distort the survey ground truth.","major_comments":[{"comment":"The definition of the moral score is ambiguous for multi-token moral judgments. The text states that 'the logit corresponding to the word appearing after the prompt is calculated,' but pairs 1, 3, and 4 are 'always justifiable' vs. 'never justifiable', 'morally good' vs. 'morally bad', and 'ethically right' vs. 'ethically wrong'—all multi-word phrases. If only the first token is scored, pair3 and pair4 have identical first tokens in both alternatives, making the score identically zero and impossible to yield the nonzero correlations reported in Tables 1–10. If the full phrase is scored as a sequence (e.g., by summing log-probabilities), the procedure is not described. The paper must specify exactly how multi-token pairs are scored, provide the corresponding formulas, and ideally release the scoring code. Until this is resolved, the central correlations, including the BLOOMZ results in Tables 9 and 10, cannot be interpreted.","section":"§4.2 (Method for Moral Score Calculation)"},{"comment":"The moral score is computed as log P(moral token) − log P(nonmoral token) with no subtraction of a country-agnostic or topic-agnostic baseline. For a pair like (right, wrong), the difference can be dominated by the marginal frequency of the two words in English or by a general positivity bias, independent of the country or topic in the prompt. Correlating this unadjusted score with survey means across country-topic pairs can therefore produce spurious alignment driven by topic-level priors or overall response tendencies. The authors should show that a control condition—for example, the same token pairs with the country and topic removed or replaced by a neutral word—does not reproduce the reported correlations. Without such a baseline, the claim that BLOOMZ 'captures certain cultural value differences' is not established.","section":"§4.2 (no baseline control)"},{"comment":"Replacing missing WVS responses (values −1, −2, −4, −5) with 0 on the original 1–10 scale is a non-neutral choice. Because 0 lies outside the valid response range and is lower than the minimum valid response (1), countries with higher nonresponse rates will have artificially deflated mean moral scores. This systematically biases the country-topic ground truth and can inflate or deflate correlations with model scores depending on the missingness pattern. The authors should either exclude nonresponses before averaging, use survey weights, or justify why a 0-imputation is unbiased; a sensitivity analysis with listwise deletion would be a minimal check.","section":"§4.1 (WVS preprocessing)"},{"comment":"For the PEW data, the paper assigns 'Depends on situation', 'Refused', and 'Don't know' responses the same code (−1) as 'morally unacceptable'. This conflates non-attitudes or missingness with a strong negative moral judgment and will pull country-topic means downward for countries with higher nonresponse or 'depends' rates. The resulting survey scores do not faithfully represent respondent opinions, undermining the ground truth against which model scores are correlated. These responses should be treated as missing (excluded) or handled in a way that does not equate them with 'morally unacceptable'.","section":"§4.1 (PEW preprocessing)"},{"comment":"The paper's headline positive result—BLOOMZ-560M's averaged correlations of 0.25/0.29 on WVS and 0.16/0.11 on PEW—is computed by averaging over five token pairs that include the ambiguous multi-token pairs 1, 3, and 4. Given the unresolved scoring ambiguity and the lack of a baseline control, these numbers cannot currently be taken as evidence that BLOOMZ approximates human moral opinions even weakly. The authors need to rerun the analysis with an unambiguous scoring rule and a baseline subtraction, and then report whether the ordering of models and the significance of the positive correlations survive.","section":"§5.2 and Tables 9–10 (BLOOMZ claim)"}],"minor_comments":[{"comment":"The dataset is described as WVS Wave 7 in Section 3.1, while Section 4.1 refers to 'Version 5' of the WVS data; the reference list also cites Round Six. Please clarify which wave/version was actually used and correct the reference.","section":"§3.1 vs. §4.1"},{"comment":"The description of the PEW response coding is easy to misread: it states that 'Not a moral issue' is assigned 0 and non-responses −1, but the previous sentence says responses were normalized between −1 and 1. Clarify whether 'morally unacceptable' itself is −1 and whether the normalization is applied after this coding.","section":"§4.1 (PEW coding)"},{"comment":"The significance stars are not tied to exact p-values or confidence intervals, and some tables have an empty p-value column. Reporting exact p-values or confidence intervals would make the correlation results more interpretable, especially given the large number of comparisons.","section":"§4.2 and Tables 1–10"},{"comment":"The text states that 'the strongest negative correlation appears with pair5 in the 'in' mode' for GPT-2 base on WVS, but Table 1 shows pair1 'in' at −0.39 and pair3 'people' at −0.28; the stated 'strongest' value does not match the table. Please reconcile the narrative with the table entries.","section":"§5.1"},{"comment":"The paper lists sample topics like 'using contraceptives' and 'getting a divorce' but does not provide the full list of the eight topics Q84A–Q84H or the exact prompt phrases used for each topic. Include the full topic list and the resulting prompt examples to make the experiments reproducible.","section":"§4.1 (PEW topics)"},{"comment":"No code or processed data is released, and the Hugging Face model versions are not pinned with commit hashes. Given the dependence on exact tokenization and normalization procedures, releasing the scoring code and processed survey means is important for verifying the results.","section":"General reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a student-led replication study with a plausible but currently under-specified measurement pipeline. The central weakness is not the level of the correlations but their validity: the multi-token scoring ambiguity and the missing-data recoding choices directly affect the quantitative claims, including the BLOOMZ result that animates the abstract and conclusion. I think the paper is fixable within its own scope—the authors can specify the scoring, add baseline controls, rerun the correlations, and report whether the conclusions change. I would not recommend rejection because the research question is timely and the survey-anchored evaluation design is appropriate in principle. However, I would ask for a rewritten methods section and a full rerun of the analysis before considering it for publication. A related concern for the editor: the paper frames itself as a replication of Ramezani and Xu (2023) and Arora et al. (2022), but it does not include a systematic comparison table with the original studies' results, which weakens its contribution as a replication study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is an honest replication that confirms what Arora et al. (2022) and Ramezani and Xu (2023) already found — open PLMs do not align strongly with cross-national moral surveys. The new model results (BLOOMZ-560M, Qwen2-0.5B) and the PEW benchmark are useful but do not amount to a new result.\n\nCredit where earned: the paper is transparent about its method and limits, shows full correlation tables with significance, and does not overclaim. The observation that token-pair choice matters more than prompt format is a useful practical note.\n\nThe load-bearing issue is Section 4.2. The moral score is defined as the difference between log-probabilities of a morally charged token and a non-morally charged token, but three of the five 'token pairs' are multiword. The phrase 'the logit corresponding to the word appearing after the prompt' does not say how multiword continuations are scored. If only the first token is used, pairs 3 and 4 have identical first tokens in both alternatives, so the score would be exactly zero — inconsistent with the reported nonzero correlations. If the full phrase is scored as a sequence, the procedure isn't described. So the BLOOMZ correlations of 0.25/0.29 and the claim that it 'shows the best performance' are not interpretable as stated.\n\nSecond, there is no baseline: no control for marginal token frequency or a general positivity bias. The stress-test note is right that correlating the raw difference with survey means can produce spurious alignment.\n\nThird, the WVS preprocessing recodes missing responses to 0, a value outside the 1–10 scale, which can pull country means down. The dataset is described as Wave 7 in §3.1, 'Version 5' in §4.1, and the reference cites 'Round six.' Inconsistent. No code or processed data is provided.\n\nBut the qualitative finding — weak and often negative correlations — holds across the many configurations reported, so the paper's main message is not in danger. The problems affect the exact effect sizes and the BLOOMZ ranking.\n\nThis paper is for readers who want an incremental replication data point. It deserves a serious referee: the flaws are fixable, and a replication with clearly specified scoring and a released pipeline would be genuinely useful. Without those fixes, I would treat the quantitative claims as provisional.","headline":"A replication that confirms known weak alignment between open PLMs and cross-national moral surveys, but the quantitative claims about BLOOMZ rest on an under-specified token-scoring step that needs fixing before effect sizes are taken seriously.","tokens_in":14689,"tokens_out":3190,"would_cite":false,"duration_ms":28993,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper finds that pre-trained language models, probed on WVS and PEW moral topics, capture some cultural value differences but align only weakly with survey-based moral standards; the multilingual BLOOMZ-560M performs best yet still…","keywords":["moral norms","cultural values","large language models","World Values Survey","PEW Global Attitudes Survey","cross-cultural alignment","log-probability probing","BLOOMZ"],"falsifier":"If correlations with survey means remain at the same magnitude when the morally charged token pairs are replaced by non-moral positive/negative word pairs such as 'good' and 'bad' or 'pleasant' and 'unpleasant', then the moral score is tracking general valence rather than country-specific moral knowledge, falsifying the paper's interpretation of its correlations.","tokens_in":13670,"feed_emoji":"🌍","tokens_out":7415,"duration_ms":60001,"temperature":0.7,"pith_summary":"This paper asks whether pre-trained language models encode the moral norms of specific countries well enough to mirror human survey responses. Using moral-topic questions from the World Values Survey and the PEW Global Attitudes Survey as ground truth, the authors probe GPT-2, OPT, BLOOMZ-560M, and Qwen2 with English prompts of the form 'In {country}, {topic} is {moral_judgment}.' They conclude that the models do capture some cultural value differences, but correlations with survey means are weak and often negative, and models tend to judge most topics as acceptable regardless of country. The best performer, BLOOMZ-560M, reaches averaged correlations of 0.25 and 0.29 on WVS and 0.16 and 0.11 on PEW, which the authors still consider far from a moderate reflection of human opinion. The study thus challenges the idea that current pre-trained language models can serve as reliable mirrors of societal moral standards.","feed_headline":"LLMs capture only weak echoes of cultural moral values","feed_subtitle":"Across 40+ countries, model scores correlate weakly with WVS and PEW survey responses; best model peaks near 0.29.","key_machinery":"The central object is the moral score, defined as the difference between the log-probability of a morally charged token and the log-probability of its opposing token, averaged over five pairs of moral judgments such as (always justifiable, never justifiable), (right, wrong), and (ethical, unethical). Concretely, $\\text{moral\\_score} = \\frac{1}{5}\\sum_i (\\log p_{\\text{moral},i} - \\log p_{\\text{nonmoral},i})$ computed from the next-token distribution after an English template prompt. This score is the bridge between models and surveys: it is correlated with the averaged normalized survey responses per country-topic pair, and the correlation coefficients are the paper's main evidence. The choice of token pair, not the prompt phrasing, drives most of the variation in results.","core_discovery":"The central claim is that pre-trained language models do not reliably encode cross-cultural moral norms. For each of 19 WVS moral topics across 55 countries and 8 PEW topics across 39 countries, the authors compute a moral score from the model's next-token log probabilities and compare it with the country-topic mean from the surveys. All four model families, GPT-2, OPT, BLOOMZ-560M, and Qwen2, produce correlations that are weak, frequently negative, and highly sensitive to which moral token pair is used. BLOOMZ-560M, the multilingual model fine-tuned for zero-shot tasks, shows the closest alignment, with averaged correlations around 0.25 to 0.29 for WVS and 0.11 to 0.16 for PEW, yet the paper concludes that even this model failed to reflect human opinions even to a moderate degree. The discovery is more a negative result and replication check: prior claims about cultural moral knowledge in language models do not survive this broader, multi-model comparison.","pith_inferences":["Not tested in the paper, the moral score may be measuring general valence rather than culturally specific morality; a direct test would replace moral adjectives with matched non-moral adjectives and check whether country correlations persist.","Since all prompts are in English, the weak performance of multilingual models could reflect an English-language prior rather than an absence of cultural knowledge; probing in each country's dominant language is a testable extension.","Averaging survey responses to country means discards within-country disagreement; models could match some population segments while failing to match the mean, which would change the interpretation of low correlations.","The paper's own discussion notes that Spearman correlations, newer models, or larger parameter variants might alter results, so the negative conclusion is contingent on this specific probe setup."],"forward_implications":["Value-sensitive applications such as content moderation or cross-cultural chatbots should not assume that models align with local moral standards; the weak and often negative correlations indicate systematic mismatch.","The relative success of BLOOMZ-560M suggests that multilingual training plus zero-shot fine-tuning is a more promising direction for cultural alignment than English-language scale alone.","Because moral token choice changes results more than prompt phrasing, any claim about a model's moral knowledge must be qualified by the exact probe vocabulary used.","The tendency of all models to produce positive moral scores for most topics implies a positivity or acceptability bias that will require explicit mitigation before models can mirror survey-style moral judgments.","The results temper earlier findings that English pre-trained models capture cultural moral norms, extending the replication to multilingual models and two large survey datasets."],"supporting_citations":[{"why":"Supplies the WVS Wave 7 ethical-values survey responses used as the human ground truth.","marker":"[Haerpfer et al., 2021]"},{"why":"Supplies the cross-cultural survey-to-prompt probing methodology this study replicates.","marker":"[Arora et al., 2022]"},{"why":"Supplies the log-probability moral-score probe and the prior finding that English PLMs capture moral norms with Western skew.","marker":"[Ramezani and Xu, 2023]"},{"why":"Provides the transformer next-token probability machinery that makes the moral score computable.","marker":"[Vaswani et al., 2017]"},{"why":"Defines GPT-2, the English monolingual model family evaluated.","marker":"[Radford et al., 2019]"},{"why":"Defines BLOOM and BLOOMZ-560M, the multilingual model family with the best observed alignment.","marker":"[Le Scao et al., 2023]"},{"why":"Defines OPT-125M and OPT-350M, the second English monolingual family evaluated.","marker":"[Zhang et al., 2022]"},{"why":"Defines Qwen2-0.5B, the other multilingual model evaluated.","marker":"[Bai et al., 2023]"}],"fun_headline_variants":["LLMs reflect cultural morals only weakly, study finds","No LLM reliably mirrors moral norms across cultures","LLMs barely track cultural morals in global surveys","Moral mirror broken: LLMs show weak cultural value alignment","Across 55 countries, LLMs barely capture moral values"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a model's log-probability gap between a morally charged token and its opposing token, measured in an English prompt, reflects the model's moral stance on that country-topic pair rather than token frequency, English-language priors, or a general positivity bias.","fun_headline_variants_meta":{"raw":{"variants":["LLMs reflect cultural morals only weakly, study finds","No LLM reliably mirrors moral norms across cultures","LLMs barely track cultural morals in global surveys","Moral mirror broken: LLMs show weak cultural value alignment","Across 55 countries, LLMs barely capture moral values"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00089,"raw_usage":{"total_tokens":3819,"prompt_tokens":903,"completion_tokens":2916,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":2839}},"tokens_in":519,"tokens_out":2916,"duration_ms":19482,"temperature":1.0,"reasoning_tokens":2839,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:49:36.645649+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If correlations with survey means remain at the same magnitude when the morally charged token pairs are replaced by non-moral positive/negative word pairs such as 'good' and 'bad' or 'pleasant' and 'unpleasant', then the moral score is tracking general valence rather than country-specific moral knowledge, falsifying the paper's interpretation of its correlations.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the WVS Wave 7 ethical-values survey responses used as the human ground truth."}],"review_version":1}