REVIEW 3 major objections 6 minor 29 references
Where did the ambiguity go? Examining how multimodal models interpret polysemous words
T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper claims that text-to-image models, given a bare polysemous word, settle on far fewer of its senses than text-generation models do, and that both fall well short of human diversity.
desk verdict A well-controlled measurement of a plausible multimodal gap, but the headline numbers rest on one LLM judge with thin validation; worth refereeing, not worth treating as settled. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The context-free polysemy probe. Each of 100 words comes with a human-verified closed inventory of candidate senses; the model sees only the bare word, and 30 independent samples per word-model are drawn. An LLM judge assigns every output to one listed sense, or to 'multiple', 'unclear', or 'other', turning ambiguity into a per-cell distribution over senses with shared inventories and judges across modalities. Normalized Shannon entropy measures how decisively a model commits to one sense; Jensen-Shannon similarity compares sense distributions between models; PCA embeds all models in one sense-distribution space to show that modality, not provider, is the dominant axis.
What would settle it
Have independent human annotators label the same set of model outputs—especially image outputs—using open-ended sense descriptions rather than the fixed inventory, and compute inter-annotator agreement and per-model entropy. If the LLM judge's labels diverge from humans mostly on non-dominant senses, image-model entropy would rise and the multimodal gap would shrink or disappear; if human labels agree with the judge, the gap is a property of the models.
Extended reading notes
Core claim
The central claim is a modality asymmetry in how foundation models express word meaning. When a polysemous word is presented alone, generated images concentrate on one dominant sense much more than generated sentences do, and neither matches human variability. The gap is consistent within every model family and across English, Turkish, and French, and it is specific to models: humans are roughly equally diverse in image and sentence framings. The paper also shows the narrowing is not a knowledge failure: models asked to state how often they would produce each sense report distributions near or above human diversity, well above what they actually generate. It further finds that diffusion-base
Load-bearing premise
The sense labels come from an LLM judge working from a fixed list of candidate senses with fallback labels; if that judge systematically assigns ambiguous or unusual outputs to the dominant sense, or bins valid rare senses as 'other', the measured entropy gap could be exaggerated.
Editorial extensions
If this is right
- If the multimodal gap is real, users of text-to-image models will see far less of a word's ambiguity than users of text models: a prompt like 'bolt' will repeatedly return a fastener, while a sentence prompt spreads across lightning, fastener, and sprinting.
- Model families' image outputs are more alike than their text outputs (mean pairwise JS similarity 0.78 vs 0.69), so swapping image providers changes the chosen dominant sense less than swapping text providers.
- Text sense diversity has declined with newer models in every family; without intervention, future text generators may approach the low diversity image models already show.
- Preference optimization can erode sense diversity (DPO drops SDXL normalized entropy from 0.25 to 0.18), so alignment choices are one lever on ambiguity collapse, though not the whole story.
- Because the narrowing appears at generation rather than at stated knowledge, fixing it likely requires changing what generation is rewarded for, not adding more knowledge about word senses.
Reading between the lines
- If the pattern extends beyond images, any modality that must commit to one reading—speech synthesis, video generation, embodied action—may silently suppress rare word senses even when the underlying model 'knows' them; the paper's stated-versus-revealed method could be ported to those modalities as a test.
- A direct intervention follows from the paper's numbers: sample generations according to the model's own stated sense prior, for example by conditioning prompts on non-dominant senses at rates proportional to the predicted distribution. If the judge is fair, entropy should rise substantially; if it does not, the stated prior is not a faithful account of the model's generative capacity.
- The finding that humans show no image-versus-text gap suggests the collapse is learned from the data and reward structure of image generation rather than inherent to visual meaning; comparing base versus aligned checkpoints, as the paper begins to do, could locate the training stage where the prior narrows.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies how text-to-image and text-generation models resolve polysemous words presented in isolation. Using 100 English words (plus 25 Turkish and French words), closed sense inventories, 30 samples per word-model cell, and a single LLM judge (GPT-5.4) for sense labeling, it compares sense-entropy across 17 image and 15 text models and against human Prolific baselines. The central claim is a multimodal gap: image models concentrate on far fewer senses (normalized entropy 0.10) than text models (0.25), and both fall well below humans (0.47–0.50). Additional results concern cross-modal convergence, preference-tuning effects, superposition in diffusion models, and a telephone-game transmission control.
Significance. If the descriptive results hold, the paper offers a clean, cross-family, cross-lingual measurement of semantic narrowing in image generation, with matched stimuli, inventories, judges, and human baselines. The design is genuinely comparative rather than anecdotal, and the telephone-game and DPO/RL ablations are useful controls. The work is not circular: the headline quantities are descriptive statistics over judged outputs, not outputs of a fitted model. The main empirical claim is falsifiable and important for understanding multimodality. However, the central dependent variable is the output of a single automatic judge under a closed inventory, and the reliability evidence is too thin to support the quantitative headline as it stands.
major comments (3)
- [§3.3, Appendix B, Appendix D] All headline Hnorm values are transforms of GPT-5.4's sense labels under a closed inventory, with fallback labels excluded. Although Appendix B says every output is labeled by both GPT-5.4 and Gemini-3.5-Flash, no agreement between the two judges is reported, and the author validation covers only 60 English images. This is load-bearing: Qwen-Image 2.0 has 29.9% fallback (15.3% unclear), and if the judge systematically resolves visually noisy images to the dominant listed sense, or bins valid but unlisted visual senses as 'other', image entropy is mechanically lowered. Report inter-judge agreement overall and by modality/language, and provide hand-validation for text and non-English outputs, or re-run the analysis on adjudicated labels.
- [§3.4.2 and Appendix D] Hnorm is computed over real senses only, with 'multiple' excluded (Appendix D: 'fallback excluded'). Yet §3.3 says a 'multiple' verdict triggers a follow-up listing the senses seen, 'so superposition is quantified rather than discarded.' The main entropy metric discards it: FLUX.2 variants have 7–11% multiple images, and Qwen-Image 2.0 has 15.3% unclear. If these outputs were assigned the listed senses, image entropy would rise. Please either integrate the follow-up senses into the sense distribution p, or show that the 0.10 vs 0.25 gap survives when multiple/unclear/other outputs are redistributed over the candidate senses.
- [§4.1] The text reports a 'paired Wilcoxon p<10−6' for the image–text entropy difference but does not specify what is paired. If the pairing is by the five provider families, p<10−6 is impossible with n=5 (minimum two-sided p=1/32). If pairing is by individual checkpoints, the pairing is not one-to-one (17 image vs 15 text models). State the pairing, the effective n, and preferably provide a model-level bootstrap or mixed-effect analysis that respects the clustering of samples within words and models.
minor comments (6)
- [§3.2] The phrase 'with recorded:' appears incomplete; presumably the comparison models are those with public training details. Please rephrase.
- [§3.3] The parenthetical in the human-baseline sentence—'(meaning vs. text models)' and '(image vs. image models)'—is confusing. Clarify which framing corresponds to which condition.
- [§3.3 vs Appendix B] The 'multiple' label is defined as '> 2 senses at once' in §3.3 but as 'two or more distinct meanings' in Appendix B. Make the definition consistent.
- [Appendix F] The appendix says '30 chains per seed' but later reports '8 out of 15 chains'; specify the number of chains and whether they are per word, per family, or per seed.
- [Throughout] No code, data, or annotation release is mentioned. For a measurement study, posting the word/sense inventories, judge prompts and outputs, and per-model Hnorm tables would substantially aid reproducibility.
- [Figure 3] The heatmap labels are very small and hard to read in print; consider larger fonts or a supplementary ordered list of models.
Circularity Check
No significant circularity: the headline numbers are descriptive statistics over judged outputs; self-citations are background, and the LLM-judge instrument is a validity concern, not a circular derivation.
full rationale
The paper's derivation chain is empirical: bare words are sampled from image/text models, each output is labeled by a judge into a closed sense inventory, and the resulting per-model distributions are summarized with normalized entropy, JS similarity, and PCA. No parameter is fitted to the data and then presented as a prediction; the stated-versus-revealed comparison (Table 2) is a separate elicitation, not a fitted function of the generation distributions. The internal self-citations (Cekinmez et al. 2025, 2026; Wu et al. 2026; Kumar et al. 2024; Brinkmann et al. 2023; Veselovsky et al. 2025) are contextual and none is load-bearing for the central multimodal-gap claim; the one interpretive appeal to Cekinmez et al. (2026) about model signatures does no derivational work. The judge-based measurement (Section 3.3, 'Using GPT 5.4 as our judge') is a reliability/validity issue rather than circularity: the paper defines the sense of an output operationally as the judge's label, with a 60-image author check, and does not pretend to derive the label from the target quantity. Concerns about missing inter-judge agreement, the closed inventory with 'other' fallback, and single-judge scoring of the human baseline are external validity threats that could affect the size of the reported gap, but they do not make any equation equivalent to its own input by construction. Accordingly, the circularity score is low, reflecting only the presence of minor, non-load-bearing self-citations.
Assumptions & free parameters
assumptions (5)
- domain assumption Candidate sense inventories in Appendix C are complete and mutually exclusive for each word in each language.
- domain assumption The LLM judge labels senses accurately and equally well across images, sentences, and human responses.
- domain assumption 30 independent samples per word-model cell give a reliable estimate of the sense distribution.
- domain assumption The bare-word image prompt and the 'use the word in a single sentence' text prompt are functionally matched; any difference reflects modality, not instruction.
- domain assumption Repeated API samples are independent and identically distributed conditional on the prompt.
Cite this review
Pith. "Pith review of Where did the ambiguity go? Examining how multimodal models interpret polysemous words." pith.science (2026). https://pith.science/paper/QWP7TKW5
@misc{pith2026260800410,
author = {Pith},
title = {Pith review of: Where did the ambiguity go? Examining how multimodal models interpret polysemous words},
year = {2026},
howpublished = {\url{https://pith.science/paper/QWP7TKW5}},
note = {Machine review of arXiv:2608.00410}
}
read the original abstract
Human language is highly polysemous. Many common words (e.g., 'bank' or 'palm') carry several distinct meanings that shape what humans communicate and imagine. Large language models (LLMs) have been shown to understand this multiplicity of meaning, but much less is known about how polysemy surfaces in other modalities such as images. We study this across 17 text-to-image and 15 text-generation models by giving each a polysemous word with no context to fix its meaning and measuring which senses are produced over many samples. We find a clear multimodal gap, where within every model family, generated images settle on far fewer senses than generated sentences (normalized entropy 0.10 vs. 0.25), and both are far less varied than what people imagine for the same words (normalized entropy 0.47). However, when we instead ask a model to list how often it would generate outputs corresponding to each possible meaning of a word, it predicts distributions that are more diverse than the actual space of outputs. These results reveal a multimodal gap in how foundation models express meaning, and how their understanding may not transfer faithfully nor equally across modalities.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
GPT-4 technical report.arXiv preprint arXiv:2303.08774,
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report.arXiv preprint arXiv:2303.08774,
-
[2]
arXiv preprint arXiv:2204.13807,
-
[8]
Gemini 2.5 Flash Image (Nano Banana) model documentation
Google DeepMind. Gemini 2.5 Flash Image (Nano Banana) model documentation. https: //ai.google.dev/gemini-api/docs/models/gemini-2.5-flash-image, 2025a. Google DeepMind. Gemini 3 Pro Image model card. https://deepmind.google/models/ model-cards/gemini-3-pro-image/, 2025b. Google DeepMind. Gemini 3.5 Flash model card. https://deepmind.google/models/ model-c...
work page 2026
-
[13]
Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,
Qwen Team. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,
-
[14]
Qwen-Image technical report.arXiv preprint arXiv:2508.02324,
Qwen Team. Qwen-Image technical report.arXiv preprint arXiv:2508.02324,
-
[15]
Qwen3.5: Towards native multimodal agents
Qwen Team. Qwen3.5: Towards native multimodal agents. https://qwen.ai/blog?id= qwen3.5, 2026a. Qwen Team. Qwen-Image-2.0 technical report.arXiv preprint arXiv:2605.10730, 2026b. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable vi...
-
[16]
Hierarchical text-conditional image generation with CLIP latents.arXiv preprint arXiv:2204.06125,
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents.arXiv preprint arXiv:2204.06125,
-
[17]
13 Oral Presentation, Scientific Understanding of Foundation Models @ COLM 2026 Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in BERTology: What we know about how BERT works.Transactions of the Association for Computational Linguistics,
work page 2026
Show all 29 references
-
[19]
Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818,
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818,
-
[20]
Gemini: A family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,
Gemini Team. Gemini: A family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,
-
[21]
Scaling monosemantic- ity: Extracting interpretable features from Claude 3 Sonnet.arXiv preprint arXiv:2605.29358,
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, et al. Scaling monosemantic- ity: Extracting interpretable features from Claude 3 Sonnet.arXiv preprint arXiv:2605.29358,
-
[22]
LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2407.21783,
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation langua...
-
[23]
Localized cultural knowledge is conserved and controllable in large language models.arXiv preprint arXiv:2504.10191,
Veniamin Veselovsky, Berke Argin, Benedikt Stroebl, Chris Wendler, Robert West, James Evans, Thomas L Griffiths, and Arvind Narayanan. Localized cultural knowledge is conserved and controllable in large language models.arXiv preprint arXiv:2504.10191,
-
[24]
SimpleAR: Pushing the frontier of autoregressive visual generation through pretraining, SFT, and RL.arXiv preprint arXiv:2504.11455,
Junke Wang, Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang, Zuxuan Wu, and Yu- Gang Jiang. SimpleAR: Pushing the frontier of autoregressive visual generation through pretraining, SFT, and RL.arXiv preprint arXiv:2504.11455,
-
[25]
Ethical and social risks of harm from language models.arXiv preprint arXiv:2112.04359,
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models.arXiv preprint arXiv:2112.04359,
-
[26]
Schr ¨odinger’s bat: Diffusion models sometimes generate polysemous words in superposition.arXiv preprint arXiv:2211.13095,
Jennifer C White and Ryan Cotterell. Schr ¨odinger’s bat: Diffusion models sometimes generate polysemous words in superposition.arXiv preprint arXiv:2211.13095,
-
[27]
Wu, Ryan Liu, Xuechunzi Bai, and Thomas L
14 Oral Presentation, Scientific Understanding of Foundation Models @ COLM 2026 Addison J. Wu, Ryan Liu, Xuechunzi Bai, and Thomas L. Griffiths. Large language models develop novel social biases through adaptive exploration. InICML,
2026
-
[28]
GLM-5.1 model card.https://huggingface.co/zai-org/GLM-5.1, 2026a
Z.ai. GLM-5.1 model card.https://huggingface.co/zai-org/GLM-5.1, 2026a. Z.ai. GLM-5.2 model card.https://huggingface.co/zai-org/GLM-5.2, 2026b. 15 Oral Presentation, Scientific Understanding of Foundation Models @ COLM 2026 A Model Inventory The complete set of evaluated model...
2026
-
[29]
imagine”) control used in the stated-versus-revealed analysis instead asks: What image comes to mind when you think of the word “{word}
Alibaba Qwen3.5 9B (Qwen Team, 2026a) Alibaba Qwen3.5 397B (Qwen Team, 2026a) Comparison models.The preference-tuning ablation (§4.3) holds the base model fixed and compares: Checkpoint Reference SDXL 1.0, base Podell et al. (2024) SDXL 1.0+Diffusion-DPO Wallace et al. (2024) ...
2024
-
[2009]
GPT-4o system card.arXiv preprint arXiv:2410.21276,
OpenAI. GPT-4o system card.arXiv preprint arXiv:2410.21276,
-
[2016]
On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258,
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258,
-
[2017]
ADAM: A diverse archive of mankind for evaluating and enhancing LLMs in biographical reasoning.arXiv preprint arXiv:2509.22991,
Jasin Cekinmez, Omid Ghahroodi, Saad Fowad Chandle, Dhiman Gupta, and Ehsaneddin Asgari. ADAM: A diverse archive of mankind for evaluating and enhancing LLMs in biographical reasoning.arXiv preprint arXiv:2509.22991,
-
[2019]
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261,
Gemini Team, Google DeepMind. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261,
-
[2021]
Pick- ing on the same person: Does algorithmic monoculture lead to outcome homogenization? InNeurIPS,
10 Oral Presentation, Scientific Understanding of Foundation Models @ COLM 2026 Rishi Bommasani, Kathleen A Creel, Ananya Kumar, Dan Jurafsky, and Percy S Liang. Pick- ing on the same person: Does algorithmic monoculture lead to outcome homogenization? InNeurIPS,
2026
-
[2022]
DeepSeekMath: Pushing the limits of mathe- matical reasoning in open language models.arXiv preprint arXiv:2402.03300,
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. DeepSeekMath: Pushing the limits of mathe- matical reasoning in open language models.arXiv preprint arXiv:2402.03300,
-
[2023]
Testing relational understanding in text-guided image generation.arXiv preprint arXiv:2208.00005,
Colin Conwell and Tomer Ullman. Testing relational understanding in text-guided image generation.arXiv preprint arXiv:2208.00005,
-
[2024]
GPT-5 system card.https://openai.com/index/gpt-5-system-card/, 2025a
12 Oral Presentation, Scientific Understanding of Foundation Models @ COLM 2026 OpenAI. GPT-5 system card.https://openai.com/index/gpt-5-system-card/, 2025a. OpenAI. Introducing our latest image generation model in the API. https://openai.com/ index/image-generation-api/, 2025...
2026
-
[2025]
Wu, and Yida Yin
Jasin Cekinmez, Ryo Mitsuhashi, Addison J. Wu, and Yida Yin. Guess the unified model: How much can we recover from generated images?arXiv preprint arXiv:2605.25254,
-
[2026]
Un-Doubling diffusion: LLM-guided disambigua- tion of homonym duplication.arXiv preprint arXiv:2509.21262,
Evgeny Kaskov, Elizaveta Petrova, Petr Surovtsev, Anna Kostikova, Ilya Mistiurin, Alexan- der Kapitanov, and Alexander Nagaev. Un-Doubling diffusion: LLM-guided disambigua- tion of homonym duplication.arXiv preprint arXiv:2509.21262,
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.