Pith. sign in

REVIEW 4 major objections 5 minor 13 references

Vision-language models respond to discourse context, but over-regularise their choices: where humans vary their sentence form, the models collapse onto a few high-probability templates.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 12:53 UTC pith:V4M4AKTM

load-bearing objection The Hungarian IS testbed is a real contribution, but the 'mode collapse' claim outruns the evidence until temperature and prompt variation are controlled. the 4 major comments →

arxiv 2605.28346 v2 pith:V4M4AKTM submitted 2026-05-27 cs.CL

When Discourse Pressures Conflict: Information Structure in Vision-Language Model Outputs

Marcell Fekete , Johannes Bjerva , Tam\'as K\'aldi This is my paper
classification cs.CL
keywords information structuretopic-focusHungarianvision-language modelsdiscourseevaluationmode collapsenatural language generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether a vision-language model, having identified the right visual content, packages that content the way a speaker would for the current discourse — putting what is already under discussion into Topic position and what is newly introduced into Focus. Because Hungarian marks Topic and Focus with dedicated word-order positions, the authors can read information-structure choices directly off the text. They compare six VLMs against native speakers producing answers to visually grounded questions, under two conditions: one where discourse, grammatical role, and definiteness point the same way, and one where discourse status conflicts with the preference to make subjects Topic and objects indefinite. The paper's central claim is that VLMs are genuinely sensitive to information structure — they answer with discourse-congruent word order most of the time — but they over-regularise that sensitivity. Humans distribute their choices across several felicitous strategies; VLMs concentrate on one template, a pattern the authors liken to mode collapse. If this is right, evaluating what a model knows about an image is not enough: evaluation must also ask whether the model produces the range of discourse-appropriate forms a language user would.

Core claim

On the paper's own terms, the discovery is a divergence between human and VLM production under conflicting discourse pressures. In the object-focus condition, where the discourse-old referent is also the grammatical subject, models match humans and topicalise the subject. In the subject-focus condition, where marking the discourse-old object as Topic would require object fronting, humans still topicalise most of the time (68.2%), while the aggregated VLMs drop to 10.4% — an 82.1-point swing versus a 13.4-point swing for humans. The same grammatical-role bias shows up in definiteness: VLMs, unlike humans, produce indefinite Focus NPs more when the Focus is an object than when it is a subject.

What carries the argument

The load-bearing machinery is Hungarian's discourse-configurational syntax, which assigns dedicated positions to Topic (first position) and Focus (immediately preverbal, diagnosed by displacement of the verb modifier). The authors build an elicited-production paradigm around this: a wh-question names one character, making it discourse-old and licensed as Topic, and elicits the other as discourse-new Focus; responses are classified into sentence types (S, V, I, O orderings) and mapped to IS-types (Top-preVF, Top-postVF, preVF, default). This gives an observable, textual read-out of information structure, letting the authors compare the distribution of human and VLM output types under aligned

Load-bearing premise

The comparison depends on humans and VLMs facing equivalent communicative conditions, but humans answered spoken questions believing a listener would use their sentence to pick the picture, while VLMs got a written system prompt asking for a short complete sentence; if that prompt narrowed the models' output style, the over-regularisation could be partly an artifact of the setup rather than an intrinsic model property.

What would settle it

Run the same VLM task with the human-style communicative framing — tell the model that a person will hear or read its sentence and must pick the right image — and measure whether subject-focus topicalisation rises toward the human 68% level. If it does, the reported collapse is a prompt artifact; if it stays near 10%, the over-regularisation is robust.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If VLMs over-regularise information structure, single-reference or majority-answer evaluation will miss a core part of discourse competence; distributional evaluation is needed.
  • Models can be fluent and semantically accurate while still failing to approximate human variability; grammaticality alone is not evidence of discourse ability.
  • The aligned/conflict distinction predicts when VLM behaviour will match humans and when it will diverge, giving a targeted testbed for future models.
  • The subject-object asymmetry in topicalisation and definiteness suggests VLM discourse choices are driven more by grammatical-role priors than by a genuine discourse model.
  • Because the pattern holds across several open and proprietary models, it likely reflects a general property of current VLM training rather than one system.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: an untested implication is that models trained with more diverse communicative text, or with human-style listener feedback, might show the missing variability; the paper does not test this.
  • Editorial extension: the same paradigm could probe other discourse-configurational languages or languages where information structure is prosodic, testing whether the over-regularisation is modality-specific.
  • Editorial extension: the mode-collapse framing suggests a concrete metric — distributional divergence between human and model IS-type distributions — that future benchmarks could report alongside accuracy; the paper stops short of proposing one.
  • Editorial extension: if the effect is driven by post-training pressure toward consistent outputs, varying temperature or decoding strategy might re-broaden the distribution; the paper did not vary these.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper investigates whether vision-language models (VLMs) produce Hungarian responses whose information structure is appropriate to the discourse context, using a psycholinguistic visual question-answering paradigm. Hungarian is chosen because Topic and Focus occupy dedicated syntactic positions, making IS choices observable in written text. Human participants and six VLMs answered questions that established either the subject or the object as discourse-old. The authors report that VLMs are sensitive to IS—they produce discourse-congruent Focus placement—but that they over-regularise relative to humans: under conflicting pressures from discourse status, grammatical role, and definiteness, humans vary their strategies, whereas VLMs converge on narrow templates, which the paper characterises as resembling mode collapse. The central evidence is distributional: VLMs show near-categorical avoidance of object topicalisation in the subject-focus condition (aggregate drop from 92.5% to 10.4% vs. humans' drop from 81.6% to 68.2%), and a subject-object asymmetry in Focus definiteness that humans do not exhibit. The paper concludes that VLM evaluation should consider not only content accuracy but the distributional shape of discourse-appropriate packaging.

Significance. If the findings hold, this is a valuable and novel contribution to VLM evaluation. The paper identifies a real blind spot—IS is rarely tested in multimodal generation—and Hungarian is a well-chosen testbed because IS is overt in word order. The experimental design controls several confounds (reversibility, animacy, verb-modifier presence), and the comparison of six models including both proprietary and open-weight systems is useful. The paper is transparent about its limitations, including demographic skew, model scope, and the unexplored role of temperature and prompt variation. The distributional perspective on evaluation is well motivated: scoring against a single reference would indeed miss the human range of acceptable IS realisations. However, the headline mode-collapse claim depends on comparing variability in VLM outputs with variability in human outputs, and the manuscript currently does not eliminate two serious confounds for that comparison: unreported decoding temperature and a human/VLM elicitation mismatch. The parser-based VLM coding also introduces a potential measurement asymmetry. These issues are fixable with additional analyses or a tempered interpretation, but

major comments (4)
  1. [§3 'Sampling and execution'; Limitations] The central claim of over-regularisation and mode collapse rests on comparing the variability of VLM outputs with human variability, but the paper never reports the decoding temperature or any sampling hyperparameters. If the models were run at low temperature or with greedy decoding, low run-to-run and within-run variability is a direct consequence of the sampling procedure, not evidence about the learned distribution. The Limitations section explicitly concedes that temperature settings were not explored. This is load-bearing: a temperature sweep across, e.g., 0.2/0.8/1.0, with variance reported per model, is needed to determine whether the narrow VLM templates are intrinsic or an artifact of decoding choices. At minimum, the mode-collapse characterisation must be conditioned on this unknown.
  2. [§3 'Adapting the procedure to VLMs'; Appendix E] The human and VLM elicitation conditions differ in a way that could directly affect response variability. Humans heard spoken instructions and were told that a later listener would use their recorded sentence to select one of four images; VLMs received a written system prompt with a nearly identical lexical instruction but no communicative-disambiguation scenario. While the prompts are worded similarly and the authors discuss the adaptation, they do not control for this mismatch. If the written prompt or the absence of a genuine listener induces a narrower register in the VLM, the observed over-regularisation could be partly an experimental artifact. A control condition—for example, the same written prompt given to human participants, or a VLM prompt that recreates the disambiguation scenario—would materially strengthen the central comparison.
  3. [§3 'Automated coding of VLM responses'; Appendix H] Human responses were manually annotated, while VLM responses were coded with an automated parser with 90.7% accuracy, and 9.9% of VLM outputs were excluded as non-categorisable. This asymmetry is a threat to the distributional comparison. If parser errors systematically map non-canonical word orders to the canonical preverbal-Focus or Top-preVF categories, the parser itself could inflate the appearance of VLM over-regularisation. The paper does not report a per-construction error analysis or a sensitivity analysis. I would ask for (a) the confusion matrix of the parser against the expert-annotated sample, (b) a manual annotation of a random VLM sample to verify that the 90.7% accuracy is not concentrated in the very categories used to diagnose over-regularisation, and (c) a worst-case re-analysis that treats excluded outputs as variability rather than errors.
  4. [§3 'Experiments with human participants'; Appendix G] Three of 51 human participants were excluded post hoc because their responses formed a distinct cluster (Top-postVF-dominant). The paper reports this transparently, and I agree that removing these participants is conservative with respect to the over-regularisation claim—it reduces human variability, making the human-VLM contrast harder to detect. Nevertheless, post-hoc exclusion based on clustering is a methodological choice that should be justified with an analysis that includes the excluded participants as a robustness check, and ideally with a pre-specified criterion for outlier exclusion. Without such a check, the exact human comparison set is difficult to replicate independently.
minor comments (5)
  1. [Table 1 / Table 3] The table notation is dense. In particular, the use of '#' and the abbreviations 'Top-preVF', 'Top-postVF', 'preVF', 'postVF' are initially confusing without the legend. Consider adding inline glosses or a separate legend directly beneath the table in the main text.
  2. [Figure 2] The labels 'Mistral' and 'GPT' appear multiple times in the upper panel without a clear mapping to the legend. A numbered or colour-matched legend directly on the figure would improve readability.
  3. [Appendix D] The power analysis yields N=55 for Mistral Small 3.1, but the paper uses N=30 for all models, including Mistral. The paper justifies this by treating Mistral as a low-competence baseline. This is reasonable, but the statistical comparisons involving Mistral should be interpreted with that under-powering in mind.
  4. [Abstract / Introduction] The phrase 'usinginformation structure' in the abstract appears to be missing a space ('using information structure'). There are several other minor spacing/rendering issues in the text; a careful proofread is recommended.
  5. [§4 'Definiteness tracks Focus...'] The statistical reporting for the definiteness analysis is compact. Reporting the per-model median differences with confidence intervals, in addition to the Wilcoxon p-values, would make the magnitude of the 'near-categorical' regularisation easier to evaluate.

Circularity Check

0 steps flagged

Empirical evaluation with no derivation-to-input reduction; minor self-citations are background only.

full rationale

The paper's central claims are empirical comparisons, not derivations whose conclusions are built into their inputs. The Hungarian IS mapping (Table 1) is imported from external literature (É. Kiss 1995, 1998, 2002) and used as a coding scheme, not fitted from the VLM or human data. The IS-type proportions and topicalisation differences are measured, not predicted from fitted parameters. The 'mode collapse' characterisation is a statistical observation about low output variability relative to humans, not a quantity defined by the analysis construction. The admitted unexamined temperature and prompt settings could threaten the validity of the variability comparison, but that is a confound, not circularity: the paper never defines VLM variability as a consequence of decoding temperature. Self-citations (e.g., Káldi and Lukács in preparation for stimuli; Káldi et al. 2020/2021 for processing background) are bibliographic or stimulus provenance references, not load-bearing premises that reduce the central result to the authors' prior claims. No uniqueness theorem, ansatz, or fitted parameter is smuggled in via self-citation. The evaluation is self-contained against its own collected human benchmark, and no equation or construction equates the conclusion with the input. Score 1 reflects only the presence of minor, non-load-bearing self-citations and the paper's own acknowledged limitations, which are correctness risks rather than circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The central claim is empirical; no parameters are fitted to data. The key axioms are the linguistic theory of Hungarian focus/topic structure and the discourse-status manipulation, both standard in the field. The paper introduces no new theoretical entities or mediators.

axioms (3)
  • domain assumption Hungarian word-order diagnostics: Topic occupies first position; preverbal Focus displaces the verb modifier (É. Kiss 1995, 1998, 2002).
    The entire IS annotation scheme (Table 1) rests on this theory of Hungarian syntax. If the diagnostics are incorrect or do not hold in the elicited register, the classification of responses into Topic/Focus patterns collapses.
  • domain assumption Wh-questions establish discourse status: the character named in the question is discourse-old (Topic candidate) and the queried character is discourse-new (Focus).
    This standard Q/A assumption is used to define the two experimental conditions. It is reasonable for human speakers, but it is assumed to hold for VLMs without direct evidence; if VLMs do not treat the named character as given, the IS-appropriateness scoring is misaligned.
  • domain assumption Definiteness–discourse status association: Topics tend to be definite; Foci tend to be indefinite (Givón 1983; É. Kiss 2002).
    This background assumption motivates the definiteness analysis (RQ2b) and the expectation that Foci should be indefinite. The paper does not test this association independently; it uses it to interpret the model–human asymmetry.

pith-pipeline@v1.3.0-alltime-deepseek · 16342 in / 10157 out tokens · 98894 ms · 2026-08-02T12:53:17.280253+00:00 · methodology

0 comments
read the original abstract

Vision-language models (VLMs) are increasingly evaluated for whether they identify the right visual content, but little is known about whether they express such content in a discourse-appropriate form. We address this research gap using information structure (IS), testing whether VLMs distinguish discourse-old Topics from discourse-new Foci in visually grounded question answering. We exploit Hungarian, a language in which Topic and Focus map onto dedicated syntactic positions, making IS choices observable in text. Comparing six VLMs with human participants, we find that models produce IS-relevant constructions, but over-regularise this sensitivity. Under the interacting pressures of discourse status, grammatical role (preference for subject Topics) and definiteness (preference for indefinite Foci), humans choose variable strategies for IS realisation. VLMs, by contrast, collapse onto narrow response templates, resembling mode collapse (Kirk et al., 2024). Our findings suggest that VLM evaluation should look beyond content accuracy to how content is packaged for the discourse.

Figures

Figures reproduced from arXiv: 2605.28346 by Johannes Bjerva, Marcell Fekete, Tam\'as K\'aldi.

Figure 1
Figure 1. Figure 1: Information structure describes how new information is established (Focus) and how already established information is managed (Topic). In our paradigm, a wh-question establishes the object (the ghost) as discourse-old and elicits the subject (the king) as discourse-new. An IS-congruent response marks the discourse-new referent as Focus and the discourse-old one as Topic; an incongruent response reverses th… view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of IS-type proportions in the object-focus (top) and subject-focus (bottom) conditions, including the IS-incongruent error condition (Top- = topicalisation, preVF = preverbal Focus, postVF = post-verbal Focus). Amber rectangles show the inter-quartile range of per-participant proportions across human participants (n = 48) with darker horizontal lines at the median. The circles show the median … view at source ↗
Figure 3
Figure 3. Figure 3: Difference in topicalisation probability in the [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Proportion of indefinite NPs in the preverbal [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The distribution of the Likert scale values for lexical (lex_likert), and morphosyntactic (m_synt_likert) [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Instructions given to (a) human participants [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Human clusters created using K-means across subject-focus and object-focus conditions. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The overall distribution of IS-types across the individual VLMs and humans. The error bars reflect variability by representing the standard deviation from the mean. preVF Top-preVF Top-postVF 0 20 40 60 80 100 Probability 1 1 2 2 3 3 4 4 5 5 5 6 6 6 7 7 OBJ_FOC preVF Top-preVF Top-postVF Sentence type 0 20 40 60 80 100 Probability 1 1 2 2 3 3 4 4 5 5 5 6 6 7 7 SUBJ_FOC model_name Claude Opus 4 (1) Gemini 2… view at source ↗
Figure 9
Figure 9. Figure 9: The distribution of IS-types across the individual VLMs and humans. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Difference in topicalisation probability in the subject-focus condition compared to the object-focus [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 2 linked inside Pith

  1. [4]

    Hungarian is a medium-resource language, and VLM fluency is less certain

    C Grammaticality Judgements The linguistic skills of LLMs and VLMs are strongest in English and other high-resource lan- guages (Joshi et al., 2020,inter alia). Hungarian is a medium-resource language, and VLM fluency is less certain. We thus evaluate VLM language skills on a set of linguistic metrics on a total of 557 samples (91-94 samples per language ...

    2020

  2. [11]

    InProceedings of the Society for Computation in Linguistics 2020, pages 409–410, New York, New York

    BLiMP: A benchmark of linguis- tic minimal pairs for English. InProceedings of the Society for Computation in Linguistics 2020, pages 409–410, New York, New York. Association for Com- putational Linguistics. Thomas Wasow

    2020

  3. [12]

    A Full Table See Table

    Verbalized sampling: How to mit- igate mode collapse and unlock llm diversity.arXiv preprint arXiv:2510.01171. A Full Table See Table

    Pith/arXiv arXiv

  4. [1972]

    Knud Lambrecht

    Functional sentence perspective: A case study from Japanese and English.Linguistic Inquiry, 3(3):269–320. Knud Lambrecht. 1994.Information Structure and Sen- tence Form: Topic, Focus, and the Mental Represen- tations of Discourse Referents, volume 71 ofCam- bridge Studies in Linguistics. Cambridge University Press, Cambridge. Bolei Ma, Yuting Li, Wei Zhou...

    1994

  5. [1995]

    An Automatic Procedure for Topic-Focus Identifica- tion.Computational Linguistics, 21(1):81–94. Jorge E. Hankamer. 1971.Constraints on deletion in syntax. Ph.D. thesis, Yale University. PhD disserta- tion. Irene Heim. 1982.The semantics of definite and in- definite noun phrases. Ph.D. thesis, University of Massachusetts, Amherst. PhD dissertation. 9 Prati...

    1971

  6. [1998]

    Katalin É

    Identificational focus versus information focus.Language, 74(2):245–273. Katalin É. Kiss. 2002.The Syntax of Hungarian, 1 edition. Cambridge University Press. Nomi Erteschik-Shir

    2002

  7. [2013]

    Harald Cramér

    Information structure: linguis- tic, cognitive, and processing approaches.WIREs Cognitive Science, 4(4):403–413. Harald Cramér. 1946.Mathematical Methods of Statis- tics. Princeton University Press, Princeton. Nicole Cuneo, Eleanor Graves, Supantho Rakshit, and Adele E. Goldberg

    1946

  8. [2014]

    InProceedings of LREC 2014, page

    The Hungarian Gigaword corpus. InProceedings of LREC 2014, page

    2014

  9. [2020]

    In Veronika Heged˝ us and Irene V ogel, editors,Approaches to Hungarian: Volume 16: Papers from the 2017 Bu- dapest Conference, volume 16, pages 73–96

    Contextual triggers of the Hungarian pre-verbal focus structure – A guided production study. In Veronika Heged˝ us and Irene V ogel, editors,Approaches to Hungarian: Volume 16: Papers from the 2017 Bu- dapest Conference, volume 16, pages 73–96. John Benjamins Publishing Company. Tamás Káldi, Ágnes Sz˝oll˝osi, and Anna Babarczy

    2017

  10. [2021]

    Katalin É

    The role of verb semantics in Hungarian verb-object order.Proceedings of the Linguistic Society of America, 6(1):54–68. Katalin É. Kiss, editor. 1995.Discourse Configura- tional Languages. Oxford University Press. Katalin É. Kiss

    1995

  11. [2023]

    In Findings of the Association for Computational Lin- guistics: EACL 2023, pages 1581–1594, Dubrovnik, Croatia

    JBLiMP: Japanese benchmark of linguistic minimal pairs. In Findings of the Association for Computational Lin- guistics: EACL 2023, pages 1581–1594, Dubrovnik, Croatia. Association for Computational Linguistics. Ekaterina Taktasheva, Maxim Bazhukov, Kirill Kon- cha, Alena Fenogenova, Ekaterina Artemova, and Vladislav Mikhailov

    2023

  12. [2024]

    InProceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9268–9299, Miami, Florida, USA

    RuBLiMP: Russian benchmark of linguistic minimal pairs. InProceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9268–9299, Miami, Florida, USA. Association for Computational Linguistics. Patricia Ward and Patrick Sturt

    2024

  13. [2025]

    Dorottya Demszky

    For GPT-4 as with humans: Information structure predicts acceptability of long- distance dependencies.Preprint, arXiv:2505.09005. Dorottya Demszky

    Pith/arXiv arXiv