Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilities

Florian Kankowski; Oliver Bott; Sina Zarriess; Torgrim Solstad

REVIEW 1 major objections 6 minor 41 references

Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilities

T0 review · 1 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that none of ten mid-size open-weight LLMs reproduces the full trio of human implicit-causality biases — coreference, coherence, and referring-expression form — and that the coreference biases seen in some larger models…

desk verdict First multi-level implicit causality benchmark for LLMs; the negative story is mostly believable, but the referring-form null depends on an unvalidated constrained decoding trick. read the letter →

arxiv 2501.12980 v1 pith:LPAXLAZH submitted 2025-01-22 cs.CL

Florian Kankowski , Torgrim Solstad , Sina Zarriess , Oliver Bott This is my paper · ORCID

classification cs.CL

keywords implicitcausalitydiscoursebiascoreferencecoherencerelationsreferringexpressionsLLMbenchmarkingpsycholinguisticsGerman

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models share the discourse biases humans show for implicit-causality verbs such as fascinate and admire. It compares ten mid-size open-weight models with human gold-standard data on three linked biases: which argument the continuation refers to (coreference), whether the continuation explains or follows the event (coherence), and whether the referring expression is a simple pronoun or a more complex form (form). The central result is negative: no model reproduces the full trio. Only the largest monolingual model, German BLOOM 6.4B, shows a statistically significant human-like coreference bias for because-continuations, the explanation bias is absent in every model, and no model shows the form bias. The authors conclude that apparent coreference sensitivity can emerge in LLMs without human-like discourse understanding, and they present the four-experiment setup as a benchmark for future models.

What carries the argument

The mechanism under test is a three-level cascade of discourse expectations associated with implicit-causality verbs: level 1 predicts a coherence relation (explanation vs. consequence), level 2 predicts the next-mention referent given that relation, and level 3 predicts the morpho-syntactic form of the referring expression given the referent. The benchmark operationalizes the cascade as four experiments: a coreference continuation task with weil and sodass prompts (Experiment 1), a coherence-relation annotation task after comma prompts (Experiment 2), and two forced-reference-style tasks measuring referring-expression form after weil and sodass (Experiments 3 and 4). LLM continuations are generated with diverse beam search, and the forced-reference tasks are emulated by constraining the first generated token to a gender-congruent personal pronoun, name, or demonstrative; human gold-standard data from the same prompt designs are re-annotated with the same automatic pipeline for comparison.

What would settle it

A reader could run Experiment 3 again with free generation followed by a post-hoc filter that keeps only continuations whose first referring expression is a gender-congruent personal pronoun, name, or demonstrative. If a human-like form bias (more pronouns for bias-congruent objects than for bias-incongruent objects) emerges in that filtered set, the paper's null result for referring-expression form would be shown to depend on the constrained decoding rather than on the models' discourse competence.

Watch

Extended reading notes

Core claim

The paper's central claim is that, across ten models from the GPT-2, mGPT, XGLM, and German BLOOM families, none exhibits human-like causal discourse when coherence and referring-expression form are considered together with coreference. In Experiment 1, only German BLOOM 6.4B shows a statistically significant human-like I-Caus coreference bias after weil 'because'; nearly all models show an I-Cons bias after sodass 'and so', but weaker than the human bias and shifted toward the object argument. In Experiment 2, no model produces explanations at a rate above 50%, and temporal relations dominate. In Experiments 3 and 4, every model prefers pronouns for subject coreference, but none shows the human pattern in which bias-congruent objects receive more pronouns than bias-incongruent objects. The paper concludes that a model can display coreference-like preferences without the coherence and form biases that form an integrated discourse-expectation system in humans.

Load-bearing premise

The load-bearing premise is that forcing the first word after the prompt to be a gender-matching pronoun, name, or demonstrative produces the same kind of continuation a human would write when asked to refer to a particular person; if this forcing changes how naturally pronouns versus names are used, the finding that models show no form bias could be a measurement artifact rather than a real deficit.

Editorial extensions

If this is right

Coreference-only tests of LLMs overestimate their discourse competence: a model can pass the coreference-bias test while failing on coherence and referring-expression form.
The coherence experiment provides a sharp behavioral probe: no tested mid-size model produced explanations above 50 percent, whereas humans produced around 80 percent after comma prompts, so this single measure separates human from model behavior cleanly.
The I-Cons bias after 'and so' is more robust and less noisy than the I-Caus bias after 'because' across models, suggesting that these models handle consequentiality better than explanation-oriented causality.
Applying the same four-experiment benchmark to larger, instruction-tuned, or closed models would show whether the missing biases emerge with scale or alignment training, and whether ChatGPT-like strong coreference behavior is accompanied by the coherence and form biases.
The hierarchical ordering of the three biases in humans predicts that models that lack the coherence and coreference biases will also lack the form bias, and the observed pattern is consistent with that ordering.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

We infer that the absence of explanation bias may partly reflect corpus statistics, since causal relations are often implicit in text while temporal relations are explicitly marked; prompting with full stops instead of commas, or probing with implicit-discourse annotations, would test this.
The constrained-decoding null result for form bias could be checked with a free-generation control: generate without constraints, then keep only continuations whose first referring expression is a pronoun, name, or demonstrative, and re-run the analysis.
The models' tendency toward female-argument coreference, opposite to the male bias often reported for LLMs, suggests that implicit-causality prompts engage gender dynamics differently from stereotype-heavy tasks; this deserves direct study.
Because I-Cons performance is stronger than I-Caus performance, the benchmark offers an empirical route to separating formal-linguistic competence from functional-causal competence in models, a distinction the paper raises in its discussion.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the quick take: this is the first benchmark that systematically tests all three implicit causality biases — coreference, coherence, referring-expression form — across a range of open-weight LLMs, and the headline negative result is mostly credible. The part I'd stake less on is the form-bias null, because it rests on an unvalidated constrained-decoding emulation.

What's genuinely new: previous LLM work on implicit causality looked almost exclusively at coreference. This paper adds coherence and referring-form, tests I-Caus and I-Cons, uses human gold-standard data from the authors' earlier psycholinguistic studies, and runs the same annotation pipeline over human and model outputs. That is real work. The finding that only German BLOOM 6.4B shows a significant human-like I-Caus coreference bias, while no model shows an explanation bias and form biases are absent, is a useful challenge to the idea that coreference sensitivity implies discourse understanding. The statistical treatment — mixed-effects regressions plus bootstrap confidence intervals — is careful and appropriate.

Soft spots, in order of importance. First, Experiments 3 and 4. The forced-reference task was emulated by constraining beam search to personal pronouns, names, and dieser/diese, with only the first token's gender constrained. Der/die demonstratives were excluded, and no filter checked whether the continuation actually referred to the intended argument. A gender-congruent sie can be plural or switch reference. Without validation against the human productions, the null form-bias result might be an artifact of the constraint set rather than a real model property. The authors acknowledge the emulation, but they don't validate it. Second, Experiment 2's automatic discourse annotation has kappa=0.68 and excludes 39.9–77.1% of LLM data. That's a lot, though the negative result is consistent across models and the authors show higher agreement after removing main clauses. Third, the promised OSF repository isn't live, so the data and scripts can't be checked yet.

None of this sinks the paper. The coherence and coreference results stand independently, and the main conclusion — mid-size open LLMs don't reproduce the human trio — is plausible. But the form-bias claim should be tempered until the emulation is validated or replaced.

Who this is for: anyone building discourse benchmarks or comparing LLM linguistic behavior with human psycholinguistic data. It deserves a serious referee. I'd send it to review, with the expectation that data are released and the forced-reference emulation is either validated or the claim is narrowed.

Referee Report

1 major / 6 minor

Summary. The paper probes ten mid-size, open-weight, non-instruction-tuned LLMs (German GPT-2, German BLOOM 350M/1.5B/6.4B, mGPT, and XGLM 0.564B/1.7B/2.9B/4.5B/7.5B) for three Implicit Causality (IC) discourse biases that are well-established in human psycholinguistic experiments: next-mention coreference bias (Experiment 1), explanation vs. other coherence relations (Experiment 2), and anaphoric form choice under forced reference (Experiments 3 and 4). Human gold-standard data come from the authors' earlier studies (Solstad and Bott 2022 for coreference and coherence; Bott and Solstad 2023 for referring forms). The central empirical findings are negative: none of the models shows the full human trio of biases. Specifically, only German BLOOM 6.4B exhibits a significant human-like I-Caus coreference verb-class effect; no model shows an explanation bias above 50%; and no model shows the human-like modulation of referring-expression form by bias congruency, although most models reproduce the general human preference for simpler forms for subject reference. The paper proposes this triad as a discourse benchmark, arguing that apparent coreference sensitivity in LLMs can exist without human-like causal discourse understanding.

Significance. If the findings survive the methodological concerns raised below, the paper makes a useful contribution: it is one of the few studies to evaluate IC in LLMs beyond the coreference bias, and it provides a concrete, multi-level benchmark that later work can apply to larger or instruction-tuned models. The authors also make several methodologically commendable choices: they use the same automatic annotation pipeline for human and LLM data where possible, report mixed-effects regression and bootstrap analyses, and run multiple models from the same family to probe size effects. The three-level model (relation -> referent -> form) is a clear and plausible way to structure the benchmark. However, the validity of the benchmark, and in particular the claim that no LLM exhibits the referring-expression form bias, currently rests on an unvalidated constrained-decoding emulation and on moderate-quality automatic annotation with substantial data exclusion in Experiment 2. These are not fatal to the coreference or coherence results per se, but they are load-bearing for the strongest negative claims.

major comments (1)

minor comments (6)

[Sections 2-4] There is a missing Section 3: the text jumps from Section 2 to a numbered but empty '3.' and then to '4. Language Model Setup'. This should be fixed.
[Sections 6-9] The papers contains numerous typos and small errors, e.g., 'constitues', 'perfomance', 'Or focus', 'futher', 'seperate', 'allmost', 'proported' (for 'purported'), and 'dependency' (for 'dependencies'). A careful copyedit is needed.
[Section 6.2] The sentence 'yielding κ = .85 and κ = .996 for coreference labeling in experiments 1 and 2, respectively' is confusing: Experiment 2 is about coherence relations, not coreference. Please clarify which annotations were compared in each experiment.
[Sections 8 and 9] The full statistical results for Experiments 3 and 4 are not reported. Only selected models are plotted, and no table gives the regression coefficients, standard errors, and p-values for all ten models. Since the central claim is that all models fail to show the form bias, full results should be provided in a table or supplementary material.
[Section 8.1] The exact prompts and the complete list of allowed token sets for the constrained decoding are not fully specified. The paper should list the exact set of allowed first tokens for each condition (which names, which pronoun forms, and which demonstrative forms) so that the experiments are reproducible.
[Section 6.3] The OSF repository is described as 'not yet published'. For a benchmark paper, data, code, and model outputs should be made available with the manuscript, or at least a stable anonymized repository should be provided for review.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper benchmarks LLM outputs against independent human behavioral data; self-citations are empirical gold standards, not premises that make the conclusions equivalent to the inputs.

full rationale

This is an empirical comparison study rather than a derivation. The human gold-standard datasets (Solstad and Bott 2022; Bott and Solstad 2023) are self-citations, but they are independent behavioral measurements collected before and outside the LLM experiments, and the same Implicit Causality biases are corroborated by a large external literature (e.g., Garvey and Caramazza 1974; Kehler et al. 2008; Hartshorne et al. 2015; Weatherford and Arnold 2021). No parameter is fitted to a subset of LLM outputs and then renamed as a prediction; the regression models are descriptive comparisons of human and model proportions. The constrained beam-search emulation of the forced-reference task (Section 8.1) restricts the first-token candidate set, but the outcome variable (proportion of personal pronouns) is not fixed by that constraint and varies across models and conditions, so the null form-bias result is not forced by construction. That emulation is a methodological validity concern about task comparability, not a circular step. The paper's negative conclusions are thus not equivalent to its inputs: they depend on the actual generated continuations, not on restating the human gold standard. Self-citations appear frequently, but they function as independently collected comparison data and are load-bearing only in the ordinary sense that any benchmark needs a reference point.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's conclusions depend on methodological choices (decoding hyperparameters, constrained generation, automatic annotation) and on the validity of the human gold standard. No parameters are fitted to make the negative results appear; the central comparison is empirical. The proposed three-level model is a framework introduced in this paper, not an independently established theory. No new entities are postulated.

free parameters (3)

diverse beam search hyperparameters = beam size=10, beam group size=10, diversity penalty lambda=0.6
Chosen based on Sieker et al. (2023), not fitted to the target result, but they determine the generated continuations and therefore all bias estimates.
constrained decoding form set (Exp 3/4) = personal pronouns (er/sie), proper names, demonstratives (dieser/diese)
Hand-selected forms allowed as the first token in the forced-reference emulation; restricts the output space and may alter form distributions.
name selection threshold = 5% gender-incongruent anaphora cutoff
Names with more than 5% incongruent pronouns in GPT-2 pilot prompts were removed; this filters stimuli and could affect coreference bias measurements.

assumptions (4)

domain assumption Implicit Causality verbs elicit three stable discourse biases in human participants: coreference, coherence, and referential form.
The benchmark rests on the psycholinguistic gold standard; supported by extensive prior literature cited in Section 2.1, but not re-established in this paper.
ad hoc to paper The three-level hierarchical model (relation to referent to form) is the correct characterization of how the biases interact.
Proposed in Section 2.1/Figure 1 to motivate the benchmark; not independently validated and used to interpret the negative results as hierarchical.
domain assumption Automatic annotation (spaCy parsing, DiMLex connective mapping) provides valid labels for LLM continuations.
Inter-annotator agreement with human labels is reported (kappa = 0.68 to 0.996), but for Experiment 2 agreement is moderate and 39.9-77.1% of data are excluded.
standard math Mixed-effects logistic regression and likelihood ratio tests are appropriate for the repeated-measures design.
Standard statistical tools used in psycholinguistics; their assumptions (e.g., random effects structure) are not explicitly verified but are conventional.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilities." pith.science (2026). https://pith.science/paper/LPAXLAZH

@misc{pith2026250112980,
  author       = {Pith},
  title        = {Pith review of: Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LPAXLAZH}},
  note         = {Machine review of arXiv:2501.12980}
}

read the original abstract

In this paper, we compare data generated with mono- and multilingual LLMs spanning a range of model sizes with data provided by human participants in an experimental setting investigating well-established discourse biases. Beyond the comparison as such, we aim to develop a benchmark to assess the capabilities of LLMs with discourse biases as a robust proxy for more general discourse understanding capabilities. More specifically, we investigated Implicit Causality verbs, for which psycholinguistic research has found participants to display biases with regard to three phenomena:\ the establishment of (i) coreference relations (Experiment 1), (ii) coherence relations (Experiment 2), and (iii) the use of particular referring expressions (Experiments 3 and 4). With regard to coreference biases we found only the largest monolingual LLM (German Bloom 6.4B) to display more human-like biases. For coherence relation, no LLM displayed the explanation bias usually found for humans. For referring expressions, all LLMs displayed a preference for referring to subject arguments with simpler forms than to objects. However, no bias effect on referring expression was found, as opposed to recent studies investigating human biases.

Figures

Figures reproduced from arXiv: 2501.12980 by the authors.

**Figure 2.** Individual I-Caus and I-Cons biases for stimulus-experiencer and experiencer-stimulus [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗

**Figure 3.** I-Caus and I-Cons biases of individual verbs for human data (Figure 3a) and the LLMs [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Distributions of discourse relations produced in continuations after prompts ending in a [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]

**Figure 5.** Figure 5: Coherence biases of stimulus-experiencer and experiencer-stimulus verbs (left and right [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]

**Figure 6.** Figure 6: Anaphoric forms produced in Bott and Solstad (2023). Data pooled together from their [PITH_FULL_IMAGE:figures/full_fig_p024_6.png]

**Figure 7.** Figure 7: Distribution of anaphoric expressions for I-Caus [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]

**Figure 8.** Figure 8: Distribution of anaphoric expressions for I-Cons [PITH_FULL_IMAGE:figures/full_fig_p027_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 21 canonical work pages

[5]

Oliver Bott, Matthias Schrumpf, Jens Michaelis, and Torgrim Solstad

Online First; DOI: 10.1177/17470218231157268. Oliver Bott, Matthias Schrumpf, Jens Michaelis, and Torgrim Solstad. Are discourse expectations modulated by being linguistically creative? a production and perception study on Implicit Causal- ity. In Proceedings of CogSci 2023, Sydney,

work page doi:10.1177/17470218231157268
[6]

To appear

doi: 10.4119/unibi/2969323. To appear. Peter Bourgonje. Shallow discourse parsing for German, volume

work page arXiv
[8]

Uncovering Constraint-Based Behavior in Neural Models via Targeted Fine-Tuning

Forrest Davis and Marten van Schijndel. Uncovering constraint-based behavior in neural models via targeted fine-tuning. arXiv preprint arXiv:2106.01207,

work page Pith review arXiv
[10]

Mono vs multilingual transformer-based mod- els: a comparison across several language tasks

Diego de Vargas Feijo and Viviane Pereira Moreira. Mono vs multilingual transformer-based mod- els: a comparison across several language tasks. arXiv preprint arXiv:2007.09757,

arXiv 2007
[13]

doi: 10.18653/v1/2023.findings-emnlp.868

Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.868. URL https://aclanthology.org/2023. findings-emnlp.868. Alan Garnham, Scarlett Child, and Sam Hutton. Anticipating causes and consequences. Journal of Memory and Language, 114:Article 104130, 2020a. ISSN 0749-596X. doi: https://doi.org/10. 1016/j.jml.2020.104130. Alan Garn...

work page arXiv 2023
[17]

The curious case of neural text degeneration

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751,

arXiv 1904
[19]

doi: 10.18653/v1/2024.starsem-1.34

As- sociation for Computational Linguistics. doi: 10.18653/v1/2024.starsem-1.34. URL https: //aclanthology.org/2024.starsem-1.34. Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom em- beddings, convolutional neural networks and incremental parsing. To appear,

doi:10.18653/v1/2024.starsem-1.34 2024
[21]

John praised Mary because he? Implicit Causality Bias and Its Interaction with Explicit Cues in LMs

ISSN 0167-5133. doi: 10.1093/jos/ffm018. Yova Kementchedjhieva, Mark Anderson, and Anders Søgaard. John praised mary because he? im- plicit causality bias and its interaction with explicit cues in lms.arXiv preprint arXiv:2106.01060,

work page Pith review arXiv

Show all 41 references

[24]

Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko

URL https://arxiv.org/abs/2112.10668. Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models. Trends in Cognitive Sciences,

arXiv
[25]

Praneeth Nemani, Yericherla Deepak Joel, Palla Vijay, and Farhana Ferdouzi Liza

URL https://arxiv.org/ abs/2408.07144. Praneeth Nemani, Yericherla Deepak Joel, Palla Vijay, and Farhana Ferdouzi Liza. Gender bias in transformers: A comprehensive review of detection and mitigation strategies. Natural Language Processing Journal, 6:100047,

arXiv
[26]

doi: https://doi.org/10.1016/j.nlp

ISSN 2949-7191. doi: https://doi.org/10.1016/j.nlp. 2023.100047. URL https://www.sciencedirect.com/science/article/pii/ S2949719123000444. Malte Ostendorff. German bloom 350m model. https://huggingface.co/malteos/ bloom-350m-german, 2022a. Oct 18, 2022 version. 34 Malte Ostend...

2023
[28]

A thorough examination of decoding methods in the era of llms

Chufan Shi, Haoran Yang, Deng Cai, Zhisong Zhang, Yifan Wang, Yujiu Yang, and Wai Lam. A thorough examination of decoding methods in the era of llms. arXiv preprint arXiv:2402.06925,

arXiv
[29]

org/abs/2204.07580

URLhttps://arxiv. org/abs/2204.07580. 35 Judith Sieker, Oliver Bott, Torgrim Solstad, and Sina Zarrieß. Beyond the bias: Unveiling the quality of implicit causality prompt continuations in language models. InProceedings of the 16th International Natural Language Generation Con...

arXiv
[30]

Torgrim Solstad and Oliver Bott

doi: 10.1080/23273798.2022.2069277. Torgrim Solstad and Oliver Bott. Implicit causality and consequentiality of action verbs. Frontiers in Language Sciences, 2(1143214),

2022
[31]

Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri `a Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint...

arXiv
[32]

doi: 10.18653/v1/P19-1164

Association for Computational Linguistics. doi: 10.18653/v1/P19-1164. URL https://aclanthology.org/P19-1164/. Manfred Stede and Carla Umbach. Dimlex: A lexicon of discourse markers for text genera- tion and understanding. In COLING 1998 V olume2: The 17th International Confere...

doi
[33]

Sep 17, 2021 version. A. J. Stewart, Martin J. Pickering, and Anthony J. Sanford. Implicit consequentiality. InProceedings of the Twentieth Annual Conference of the Cognitive Science Society, pages 1031–1036, Mah- wah, NJ,

2021
[35]

Diverse beam search: Decoding diverse solutions from neural se- quence models

36 Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural se- quence models. arXiv preprint arXiv:1610.02424,

arXiv
[37]

doi: 10.18653/v1/2020

Association for Computational Linguistics. doi: 10.18653/v1/2020. emnlp-main.16. URL https://aclanthology.org/2020.emnlp-main.16. Kathryn C. Weatherford and Jennifer E. Arnold. Semantic predictability of implicit causality can affect referential form choice. Cognition, 214:104759,

2020 doi
[38]

doi: https: //doi.org/10.1016/j.cognition.2021.104759

ISSN 0010-0277. doi: https: //doi.org/10.1016/j.cognition.2021.104759. Taylor Webb, Keith J Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1526–1541,

2021
[39]

Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Aky ¨urek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim

URL https://arxiv.org/abs/2310.15113. Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Aky ¨urek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arX...

arXiv
[40]

How well do large language models perform in arithmetic tasks? arXiv preprint arXiv:2304.02015,

Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. How well do large language models perform in arithmetic tasks? arXiv preprint arXiv:2304.02015,

arXiv
[41]

This isn’t the bias you’re looking for: Implicit causality, names and gender in german language models

Sina Zarrieß, Hannes Gr ¨oner, Torgrim Solstad, and Oliver Bott. This isn’t the bias you’re looking for: Implicit causality, names and gender in german language models. In Proceedings of the 18th Conference on Natural Language Processing (KONVENS 2022), pages 129–134,

2022
[42]

URL https://aclanthology.org/2021.acl-long.90

18653/v1/2021.acl-long.90. URL https://aclanthology.org/2021.acl-long.90. Mingyu Zong and Bhaskar Krishnamachari. A survey on gpt-3. arXiv preprint arXiv:2212.00857,

2021 arXiv
[1980]

Ronen Eldan and Yuanzhi Li

doi: 10.1080/14640748008401161. Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?,

doi
[1990]

Shiva Upadhye, Leon Bergen, and Andrew Kehler. Predicting reference: What do language models learn about discourse models? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 977–982,

2020
[2001]

URL http://dx.doi.org/10.1075/hcp.8.04ari

doi: 10.1075/hcp.8.04ari. URL http://dx.doi.org/10.1075/hcp.8.04ari. Jennifer E. Arnold. How speakers refer: The role of accessibility. Language and Linguistics Compass, 4(4):187–203,

doi
[2006]

31 Forrest Davis and Marten van Schijndel

doi: 10.1080/ 01690960500199763. 31 Forrest Davis and Marten van Schijndel. Discourse structure interacts with reference but not syntax in neural language models. arXiv preprint arXiv:2010.04887,

2010 arXiv
[2008]

Assessing bert’s syntactic abilities

Yoav Goldberg. Assessing bert’s syntactic abilities. arXiv preprint arXiv:1901.05287,

1901 arXiv
[2010]

doi: 10.1016/j.jml.2009.09.001

ISSN 0749-596X. doi: 10.1016/j.jml.2009.09.001. Changjiang Gao, Shujian Huang, Jixing Li, and Jiajun Chen. Roles of scaling and instruction tun- ing in language perception: Model vs. human attention. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Associ...

2009 doi
[2011]

doi: 10.3758/s13428-010-0023-2

ISSN 1554-3528. doi: 10.3758/s13428-010-0023-2. Kumiko Fukumura and Roger P.G. van Gompel. Choosing anaphoric expressions: Do people take into account likelihood of reference? Journal of Memory and Language, 62(1):52–66,

doi
[2013]

Implicitness of discourse relations

Fatemeh Torabi Asr and Vera Demberg. Implicitness of discourse relations. In Proceedings of COLING 2012, pages 2669–2684, Mumbai,

2012
[2015]

Robert D

doi: 10.1080/23273798.2015.1008524. Robert D. Hawkins, Takateru Yamakoshi, Thomas L. Griffiths, and Adele E. Goldberg. Investigating representations of verb bias in neural language models,

2015
[2016]

Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. Learning which features matter: RoBERTa acquires a preference for linguistic generalizations (eventu- ally). In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors,Proceedings of the 2020 Confer...

2020
[2017]

Towards reasoning in large language models: A survey

Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403,

arXiv
[2019]

Xudong Hong, Margarita Ryzhova, Daniel Biondi, and Vera Demberg. Do large language models and humans have similar behaviours in causal inference with script knowledge? In Danushka Bollegala and Vered Shwartz, editors, Proceedings of the 13th Joint Conference on Lexical and Com...

2024
[2020]

doi: 10.18653/v1/2020.coling-main.107

International Committee on Computational Linguistics. doi: 10.18653/v1/2020.coling-main.107. URL https://aclanthology.org/2020. coling-main.107. Mira Ariel. Accessing noun-phrase antecedents. Routledge, London,

2020 doi
[2021]

Dynabench: Rethinking bench- marking in nlp

Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. Dynabench: Rethinking bench- marking in nlp. arXiv preprint arXiv:2104.14337,

arXiv
[2022]

Sorting through the noise: Testing robustness of information processing in pre-trained language models

Lalchand Pandia and Allyson Ettinger. Sorting through the noise: Testing robustness of information processing in pre-trained language models. arXiv preprint arXiv:2109.12393,

arXiv
[2023]

Deep rnns encode soft hierarchical syntax.arXiv preprint arXiv:1805.04218,

Terra Blevins, Omer Levy, and Luke Zettlemoyer. Deep rnns encode soft hierarchical syntax.arXiv preprint arXiv:1805.04218,

arXiv
[2024]

Morton A

URL https://arxiv.org/abs/2409.05653. Morton A. Gernsbacher and D.J. Hargreaves. Accessing sentence participants: The advantage of first mention. Journal of Memory and Language, 27(6):699–717,

arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [5]

Oliver Bott, Matthias Schrumpf, Jens Michaelis, and Torgrim Solstad

Online First; DOI: 10.1177/17470218231157268. Oliver Bott, Matthias Schrumpf, Jens Michaelis, and Torgrim Solstad. Are discourse expectations modulated by being linguistically creative? a production and perception study on Implicit Causal- ity. In Proceedings of CogSci 2023, Sydney,

work page doi:10.1177/17470218231157268

[2] [6]

To appear

doi: 10.4119/unibi/2969323. To appear. Peter Bourgonje. Shallow discourse parsing for German, volume

work page arXiv

[3] [8]

Uncovering Constraint-Based Behavior in Neural Models via Targeted Fine-Tuning

Forrest Davis and Marten van Schijndel. Uncovering constraint-based behavior in neural models via targeted fine-tuning. arXiv preprint arXiv:2106.01207,

work page Pith review arXiv

[4] [10]

Mono vs multilingual transformer-based mod- els: a comparison across several language tasks

Diego de Vargas Feijo and Viviane Pereira Moreira. Mono vs multilingual transformer-based mod- els: a comparison across several language tasks. arXiv preprint arXiv:2007.09757,

arXiv 2007

[5] [13]

doi: 10.18653/v1/2023.findings-emnlp.868

Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.868. URL https://aclanthology.org/2023. findings-emnlp.868. Alan Garnham, Scarlett Child, and Sam Hutton. Anticipating causes and consequences. Journal of Memory and Language, 114:Article 104130, 2020a. ISSN 0749-596X. doi: https://doi.org/10. 1016/j.jml.2020.104130. Alan Garn...

work page arXiv 2023

[6] [17]

The curious case of neural text degeneration

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751,

arXiv 1904

[7] [19]

doi: 10.18653/v1/2024.starsem-1.34

As- sociation for Computational Linguistics. doi: 10.18653/v1/2024.starsem-1.34. URL https: //aclanthology.org/2024.starsem-1.34. Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom em- beddings, convolutional neural networks and incremental parsing. To appear,

doi:10.18653/v1/2024.starsem-1.34 2024

[8] [21]

John praised Mary because he? Implicit Causality Bias and Its Interaction with Explicit Cues in LMs

ISSN 0167-5133. doi: 10.1093/jos/ffm018. Yova Kementchedjhieva, Mark Anderson, and Anders Søgaard. John praised mary because he? im- plicit causality bias and its interaction with explicit cues in lms.arXiv preprint arXiv:2106.01060,

work page Pith review arXiv