Smoothie-Qwen: Post-Hoc Smoothing to Reduce Language Bias in Multilingual LLMs

Jemin Kim; JungYup Lee; Sang Park; Seungjae Lee; SeungWon Ji

REVIEW 4 major objections 6 minor 12 references

Smoothie-Qwen: Post-Hoc Smoothing to Reduce Language Bias in Multilingual LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Smoothie-Qwen claims that scaling lm head weights of high-risk Chinese tokens, using a non-linear smoothing function, suppresses unintended Chinese generation by over 95% while preserving Korean task accuracy.

desk verdict A clean, reproducible token-weight scaling trick with a narrower practical effect than claimed; the 95% figure is an artifact of the elicitation set, and the key logit-sign assumption is unverified. read the letter →

arxiv 2507.05686 v1 pith:JEXNKHTG submitted 2025-07-08 cs.CL

SeungWon Ji , Jungyup Lee , Jemin Kim , Sang Park , SeungJae Lee This is my paper

classification cs.CL

keywords languagebiasconfusionmultilingualLLMspost-hocsmoothinglmheadweightscalingtokenriskscoringQwen2.5-CoderKMMLU

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Smoothie-Qwen, a training-free post-hoc method that reduces language bias in multilingual LLMs by scaling down the output-layer weights of tokens identified as belonging to an undesired language. Applied to the Qwen2.5-Coder-14B-Instruct model, it suppresses unintended Chinese generation by over 95% at the recommended hyperparameters, while KMMLU Korean task accuracy stays essentially unchanged. The authors argue that token-level logit adjustment is a practical, sufficient control for language confusion without retraining. The method matters because deployed multilingual models often answer in a dominant language, and a cheap fix that does not touch training data could improve reliability for non-dominant-language users.

What carries the argument

The central mechanism is the non-linear smoothing function $S = 1 - (1 - \min\_scale) \times \frac{\log(1 + (\text{smoothness} - 1) \times \text{risk})}{\log(\text{smoothness})}$, which maps each token's risk score in [0,1] to a weight-scaling factor in [min_scale,1]. Target tokens are found by Unicode range scanning plus n-gram sampling for broken subword tokens; each gets a risk score, and the lm head weight vector is multiplied by S before generation. The authors rely on the empirical observation that high-risk tokens overwhelmingly have positive logits, so shrinking their weights lowers their softmax probability.

What would settle it

Measure the lm head logits of the 27,610 Chinese and broken tokens across a diverse set of prompts: if a substantial fraction are negative, and the scaled model increases their sampled frequency, the central mechanism fails. A simpler check is to run a Chinese-language task suite and see whether accuracy collapses after smoothing.

Watch

Extended reading notes

Core claim

Smoothie-Qwen's central claim is that language bias can be mitigated by a one-time, context-free scaling of the lm head weight vectors of high-risk target-language tokens, using a logarithmic smoothing curve. The paper reports that with min_scale=0.5 and smoothness=10.0, the Chinese suppression metric rises from 0.190 to 0.950, while Korean multiple-choice accuracy (acc cs, acc ie) stays flat at 0.710 and 0.395. Qualitative examples show the model substitutes pinyin, switches to English, or avoids the token entirely, rather than blocking output outright.

Load-bearing premise

The method works only if almost every token flagged as Chinese has a positive pre-softmax score, so shrinking its weight always lowers its probability; if many such tokens had negative scores, the smoothing could make them more likely and the whole mechanism would backfire.

Editorial extensions

If this is right

Owners of deployed Qwen-style models can apply Smoothie-Qwen at load time with no retraining and no access to training data.
At the recommended settings, the suppression rate for unintended Chinese generation rises from 0.190 to 0.950, while Korean task accuracy is preserved.
The method is soft: suppressed tokens are downscaled, not zeroed, so the model can still produce them when needed.
Because the method never touches other tokens' weights, the paper expects general reasoning capabilities to remain intact.
Tuning min_scale and smoothness controls the trade-off between suppression strength and expressive capacity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the positive-logit dominance claim holds broadly, the same weight-scaling recipe could be transferred to other language pairs and to non-language token classes such as profanity or style markers, but each transfer would need its own risk-score calibration.
Because smoothing is static and context-blind, a legitimate request to translate into Chinese would also be suppressed; a dynamic gating controller is a natural testable extension.
The Korean KMMLU results do not test any task that actually requires Chinese, so the accuracy-preservation claim is only established for non-Chinese-required tasks; a Chinese QA benchmark would be a direct stress test.
The n-gram risk score for broken tokens is a heuristic; a corpus-based co-occurrence estimate could improve coverage without changing the core mechanism.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, this is a simple, reproducible post-hoc trick: multiply the lm-head weight vectors of Chinese-range tokens by a scalar between 0.5 and 1 based on a logarithmic risk score, and you can suppress Chinese output on a Qwen2.5-Coder model. Second, the headline 95% suppression number is real only for their synthetic 'elicit Chinese translation' set; on the KMMLU generation metrics the model already produces almost no Chinese (0.995/0.990 at baseline), so the method's practical effect on real confusion is much smaller than the abstract implies.

What's new: the specific combination of Unicode-range token flags, an n-gram heuristic for broken tokens, and a non-linear scaling curve is a modest twist on token-level logit adjustment. The paper ships code and pre-modified models, which is welcome. The algorithm is clearly specified and the KMMLU accuracy numbers are stable across hyperparameters, suggesting no obvious catastrophic forgetting.

Soft spots. The positive-logit assumption in Section 3.4.1 is asserted but not demonstrated. If a sizable fraction of Chinese tokens have negative logits at inference time, scaling them toward zero can actually raise their softmax probability. Without a histogram or at least a fraction, this is a load-bearing gap; the method could be working for reasons the authors don't understand, or could fail on other models/languages. No error bars or seed variation are reported, so it's impossible to tell whether the chin_prom differences between hyperparameter settings are noise. There are no baselines--no comparison against logit adjustment, vocabulary filtering, or prompt-based debiasing--so the incremental benefit is unclear. The 'preserving task accuracy' claim is tested only on Korean multiple-choice; there is no task where Chinese is actually required, and the static nature (which blocks legitimate Chinese) is acknowledged but not quantified. The custom elicitation dataset is not released, so the main metric is not independently checkable.

Overall: this is a legitimate engineering contribution with a clean description and useful code, but the evidence as presented overclaims. It deserves a serious referee, not a desk reject, because the core mechanism is testable and the code is public. A revision with released data, baselines, error bars, a direct test of the logit-sign distribution, and one task requiring the suppressed language would turn this into a solid, citable result. I'd send it to review, flagging those issues. I wouldn't cite it in the next year unless the revised evidence holds up.

Referee Report

4 major / 6 minor

Summary. The paper proposes Smoothie-Qwen, a post-hoc method that modifies the lm head weights of a pretrained multilingual LLM by downscaling tokens identified as Chinese via Unicode ranges and subword analysis, using a non-linear smoothing function with two hyperparameters (min_scale, smoothness). The method is applied to Qwen2.5-Coder-14B-Instruct. The authors report that with recommended settings (min_scale=0.5, smoothness=10.0), unintended Chinese generation is reduced by over 95% on a custom 1,000-prompt Chinese elicitation dataset, while KMMLU Korean task accuracy changes only slightly (acc_cs from 0.715 to 0.710, acc_ie from 0.385 to 0.395). The paper includes qualitative examples and lists limitations including static application, heuristic risk estimation, and manual hyperparameter tuning.

Significance. If the central claim holds, Smoothie-Qwen would be a lightweight, training-free approach to language-controllability that could be applied to deployed models without retraining. The code and ready-to-use models are publicly available, which supports reproducibility. However, the evaluation has critical weaknesses: the primary suppression metric is a synthetic elicitation proxy, the positive-logit assumption underlying the mechanism is asserted without evidence, baselines and error bars are absent, and accuracy preservation rests on a single Korean multiple-choice benchmark with no Chinese-required task. These issues make the magnitude and generality of the claimed 95% reduction uncertain, though the method itself is simple and plausible. The paper would be significantly strengthened by supplying the missing analyses and by framing the claims more carefully.

major comments (4)

[Section 3.4.1] The central mechanism rests on the claim that 'high-risk tokens—such as those from the target language—overwhelmingly have positive logits,' but no data are shown. If a substantial fraction of Chinese tokens have negative lm-head logits, scaling them toward zero increases their softmax probability, which could counteract the suppression. Provide a histogram or table of the logit distribution for the 26,153 target tokens, the fraction with positive logits, and the resulting change in softmax probabilities; alternatively, restrict scaling to positive-logit tokens or analyze the consequences of the negative-logit case. This is load-bearing because the method's effectiveness depends on it.
[Section 4.1.2 (chin_prom) and Algorithm 1] The primary suppression metric, chin_prom, measures the proportion of 1,000 synthetic prompts whose responses contain no Chinese characters. Because the algorithm assigns a risk score of 1.0 to every token in the Chinese Unicode range and then scales those tokens' weights, this metric directly measures the intended effect of the construction rather than an independent reduction in language bias. The baseline chin_prom of 0.190 also shows that 19% of prompts already produce no Chinese before smoothing, so the reported 95% improvement is a marginal change in a proxy, not a measurement of real contamination rates. Report token-level statistics (e.g., proportion of Chinese tokens in generations before/after) and release the elicitation dataset so readers can assess representativeness.
[Table 1 and Table 2] At baseline (min_scale=1.0), the generative Korean KMMLU metrics are already chin_cs=0.995 and chin_ie=0.990, indicating that the model produces almost no Chinese in these Korean-domain generation tasks. The claimed reduction of 'unintended Chinese output by over 95%' therefore rests entirely on the custom elicitation dataset, not on a scenario where Chinese contamination actually occurs at scale. The authors should evaluate on a setting where unintended Chinese generation is frequent (e.g., a multilingual translation or general instruction-following benchmark) or explicitly acknowledge that the method's measured benefit is limited to the synthetic elicitation setup.
[Section 4.2] The evaluation lacks baseline comparisons and statistical robustness. There is no comparison with simpler alternatives such as hard token blocking, logit adjustment without smoothing, or prompt-based interventions. No error bars, confidence intervals, or seed variation are reported, so the accuracy differences in Tables 1 and 2 (e.g., acc_ie changing from 0.385 to 0.395) cannot be distinguished from noise. Moreover, the claim that 'task accuracy is preserved' is supported only by Korean KMMLU; no Chinese-required task is tested, despite the method's explicit suppression of Chinese tokens. Add at least one baseline, repeated runs, and an evaluation of the model's ability to generate Chinese when explicitly requested (e.g., translation to Chinese).

minor comments (6)

[Abstract] The phrase 'proposeSmoothie-Qwen' is missing a space; it should read 'propose Smoothie-Qwen.'
[Section 5.2] The heading 'Extension to Other Bianses and Styles' contains a typo; it should be 'Extension to Other Biases and Styles.'
[Section 3.3] The phrase 'e.g., 1,457n' in the description of n-gram sampling is unclear; it should specify the exact number of possible combinations, such as '1,457 choose n' or '1,457^n'.
[Equation (1)] The smoothing function in Equation (1) would benefit from explicit punctuation and a clear definition of the domain of each variable, including the allowed range of smoothness (e.g., smoothness > 1) and the behavior at the boundaries risk_score = 0 and risk_score = 1.
[Section 3.2] The Unicode range U+4E00 to U+9FFF covers the basic CJK Unified Ideographs but excludes CJK Extension A and other supplementary planes; the paper should acknowledge that tokens containing such characters are not flagged, and discuss whether this omission affects the results.
[Section 4.1.2] The custom Chinese elicitation dataset is not released, and the description does not specify the number of n-gram samples used in the broken-token risk estimation, nor the exact construction procedure beyond '26 categories.' Provide these details or release the dataset to allow replication.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the metric/intervention alignment is an experimental design choice, not a fitted identity, and the accuracy result is independent.

full rationale

The paper's derivation chain is self-contained and does not reduce to its inputs by equation or self-citation. Step 1 flags target tokens by Unicode range, Step 2 assigns risk 1.0 to them, and Eq. (1) scales the lm-head weights; the evaluation then measures Chinese-character presence in outputs. Although the direction of suppression is aligned with the intervention by definition, the reported magnitudes (e.g., chin_prom 0.950 at min_scale=0.5) and the stable KMMLU accuracies are empirical measurements, not consequences of an equation, a fitted parameter, or a prediction that is equivalent to its construction. There is no load-bearing self-citation: the cited results are external baselines and benchmarks. The weakest load-bearing premise, Section 3.4.1's assertion that 'Empirical analysis confirms that high-risk tokens ... overwhelmingly have positive logits,' is an unsupported empirical claim about model internals that could be checked by inspecting logits; it is an evidence gap and a correctness risk, not circularity. Similarly, the synthetic chin_prom proxy and the lack of a Chinese-required benchmark are external-validity concerns, not circular steps.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

No new entities are postulated. The method's free parameters (min_scale, smoothness, n-gram sampling size) are tuned to achieve the reported suppression. The key untested assumption is the positive-logit dominance of Chinese tokens.

free parameters (4)

min_scale = 0.5
Chosen to achieve 95% suppression on chin_prom; lower values yield diminishing returns (Table 1).
smoothness = 10.0
Chosen as moderate; higher values increase suppression but are not necessary (Table 2).
n_gram sample count = 100
Number of random token pairs used to estimate broken-token risk; no justification provided (Section 3.3).
n_gram size = 2, 3, 4 (configurable)
The N-gram size for broken token risk estimation is a configurable parameter without a fixed value in experiments.

assumptions (3)

domain assumption High-risk tokens predominantly have positive logits
Section 3.4.1 asserts this based on 'empirical analysis' but no data or figure is provided. If false, weight scaling could increase the softmax probability of negative-logit Chinese tokens.
domain assumption Unicode range is a sufficient proxy for language identity
Section 3.2 flags tokens within U+4E00-U+9FFF as Chinese; this ignores other Chinese characters (e.g., extensions) and overlaps with Japanese Kanji, potentially causing false positives.
domain assumption lm head weight scaling modifies generation probabilities as intended
The effect is mediated by softmax and sampling; the paper assumes lower weights translate to lower generation frequency, which holds only on average over contexts.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Smoothie-Qwen: Post-Hoc Smoothing to Reduce Language Bias in Multilingual LLMs." pith.science (2026). https://pith.science/paper/JEXNKHTG

@misc{pith2026250705686,
  author       = {Pith},
  title        = {Pith review of: Smoothie-Qwen: Post-Hoc Smoothing to Reduce Language Bias in Multilingual LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JEXNKHTG}},
  note         = {Machine review of arXiv:2507.05686}
}

read the original abstract

Multilingual large language models (LLMs) often exhibit language confusion, a tendency to generate responses in a dominant language irrespective of the prompt's language. To address this, we propose Smoothie-Qwen, a lightweight, post-hoc method that mitigates language bias without retraining. This technique selectively adjusts token-level output probabilities to effectively suppress undesired language generation. Applied to the Qwen model, our method reduces unintended Chinese output by over 95% while preserving task accuracy on multilingual benchmarks. This work provides a practical and efficient solution for enhancing the language controllability of LLMs, making them more reliable for global applications.

Figures

Figures reproduced from arXiv: 2507.05686 by the authors.

**Figure 2.** Visualization of the non-linear smoothing function, illustrating the relationship between the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Visual comparison of lm head weights before (left) and after (right) the smoothing transformation. This figure shows a specific slice of token indices (98,000-100,000) that contains a mix of Chinese and non-Chinese tokens. In the ”Modified Model” plot, the weights corresponding to Chinese tokens (which have a high risk score) are scaled towards zero. This causes both the originally high (hot colors) and low (cool co… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 4 canonical work pages

[1]

Bias and fairness in large language models: A survey

Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics, 50 0 (3): 0 1097--1179, 2024 a

work page 2024
[2]

Self-Debiasing Large Language Models: Zero-Shot Recognition and Reduction of Stereotypes

Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Tong Yu, Hanieh Deilamsalehy, Ruiyi Zhang, Sungchul Kim, and Franck Dernoncourt. Self-debiasing large language models: Zero-shot recognition and reduction of stereotypes. arXiv preprint arXiv:2402.01981, 2024 b

work page Pith review arXiv 2024
[3]

The language model evaluation harness, 07 2024

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The languag...

arXiv 2024
[4]

The llama 3 herd of models

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

arXiv 2024
[5]

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024

arXiv 2024
[6]

BanStereoSet: A Dataset to Measure Stereotypical Social Biases in LLMs for Bangla

Mahammed Kamruzzaman, Abdullah Al Monsur, Shrabon Das, Enamul Hassan, and Gene Louis Kim. Banstereoset: A dataset to measure stereotypical social biases in llms for bangla. arXiv preprint arXiv:2409.11638, 2024

work page Pith review arXiv 2024
[7]

Understanding and mitigating language confusion in llms

Kelly Marchisio, Wei-Yin Ko, Alexandre B \'e rard, Th \'e o Dehaze, and Sebastian Ruder. Understanding and mitigating language confusion in llms. arXiv preprint arXiv:2406.20052, 2024

arXiv 2024
[8]

Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies

Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:2308.03188, 2023

arXiv 2023

Show all 12 references

[9]

Kmmlu: Measuring massive multitask language understanding in korean

Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. Kmmlu: Measuring massive multitask language understanding in korean. arXiv preprint arXiv:2402.11548, 2024

2024 arXiv
[10]

Qwen3 technical report

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

2025 arXiv
[11]

Harnessing large language models as post-hoc correctors

Zhiqiang Zhong, Kuangyu Zhou, and Davide Mottin. Harnessing large language models as post-hoc correctors. arXiv preprint arXiv:2402.13414, 2024

2024 arXiv
[12]

Evaluating and mitigating gender bias in generative large language models

Hanqing Zhou, Diana Inkpen, and Burak Kantarci. Evaluating and mitigating gender bias in generative large language models. INTERNATIONAL JOURNAL OF COMPUTERS COMMUNICATIONS & CONTROL, 19 0 (6), 2024

2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Bias and fairness in large language models: A survey

Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics, 50 0 (3): 0 1097--1179, 2024 a

work page 2024

[2] [2]

Self-Debiasing Large Language Models: Zero-Shot Recognition and Reduction of Stereotypes

Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Tong Yu, Hanieh Deilamsalehy, Ruiyi Zhang, Sungchul Kim, and Franck Dernoncourt. Self-debiasing large language models: Zero-shot recognition and reduction of stereotypes. arXiv preprint arXiv:2402.01981, 2024 b

work page Pith review arXiv 2024

[3] [3]

The language model evaluation harness, 07 2024

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The languag...

arXiv 2024

[4] [4]

The llama 3 herd of models

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

arXiv 2024

[5] [5]

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024

arXiv 2024

[6] [6]

BanStereoSet: A Dataset to Measure Stereotypical Social Biases in LLMs for Bangla

Mahammed Kamruzzaman, Abdullah Al Monsur, Shrabon Das, Enamul Hassan, and Gene Louis Kim. Banstereoset: A dataset to measure stereotypical social biases in llms for bangla. arXiv preprint arXiv:2409.11638, 2024

work page Pith review arXiv 2024

[7] [7]

Understanding and mitigating language confusion in llms

Kelly Marchisio, Wei-Yin Ko, Alexandre B \'e rard, Th \'e o Dehaze, and Sebastian Ruder. Understanding and mitigating language confusion in llms. arXiv preprint arXiv:2406.20052, 2024

arXiv 2024

[8] [8]

Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies

Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:2308.03188, 2023

arXiv 2023