Pith. sign in

REVIEW 3 major objections 6 minor 23 references

Un-considering Contextual Information: Assessing LLMs' Understanding of Indexical Elements

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large language models interpret the English indexical 'I' almost perfectly but systematically misfire on 'you', 'here', and 'tomorrow', and direct quotation helps only some of these words.

desk verdict Useful first English indexical benchmark, but the appendix inverts the answer key for 'you' and 'tomorrow', and that threatens exactly the two headline failures. read the letter →

arxiv 2506.01089 v1 pith:DLZITXZ7 submitted 2025-06-01 cs.CL cs.AI

Metehan Oguz , Yavuz Bakman , Duygu Nur Yaldiz This is my paper
classification cs.CLcs.AI
keywords indexicalelementscoreferenceresolutionlargelanguagemodelsdirectquotationcontextprimeshiftEnglishDatasetdeicticexpressions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models resolve coreference well for nouns and third-person pronouns, but this paper asks whether that skill extends to indexical elements—words like 'I', 'you', 'here', and 'tomorrow' whose referent depends on who is speaking, where, and when. Using a new 1,600-item English multiple-choice benchmark, the paper finds that accuracy is not uniform: GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and DeepSeek-V3 all interpret 'I' at near-ceiling levels, while 'you', 'here', and 'tomorrow' trigger systematic failures, especially when the surrounding context primes the wrong reading. It also finds that direct quotation—which shifts an indexical into the reported context—raises accuracy for 'here' but lowers it for 'you' and 'tomorrow'. The paper matters because it shows that LLM coreference competence is fragile: models can be lured away from the grammatically determined reading by linguistically irrelevant story cues.

What carries the argument

The load-bearing object is the English Indexical Dataset, built from 100 sentence templates per indexical, each transformed into four conditions: quotation versus non-quotation crossed with a shifted context prime (which makes the reported reading feel natural) versus a non-shifted prime (which makes the actual-utterance reading feel natural). The ground-truth rule is the semantic distinction the paper tests: in non-quoted sentences an indexical refers to the actual speech-act coordinates (who is speaking, where, and when), whereas inside direct quotation it shifts to the reported context. This two-by-two design lets the paper separate the effect of syntactic cues (quotation) from the effect of pragmatically irrelevant context (the prime), and the accuracy breakdown across the four cells is the evidence for every claim.

What would settle it

Re-annotate the 1,200 unchecked items with two independent human annotators and recompute every accuracy score on items where both annotators agree. If agreement is low in quotation conditions or in 'here' and 'tomorrow' items, or if the recomputed pattern no longer shows quotation helping 'here' while hurting 'you' and 'tomorrow', the paper's central distinction fails.

Watch

Extended reading notes

Core claim

The central discovery is that LLM performance on indexical interpretation is not uniform across indexical types or sentence types. On non-quoted sentences, all four models choose the correct non-shifted reading of 'I' about 99% of the time, but accuracy collapses for 'you', 'here', and 'tomorrow' when the context primes the shifted reading; for 'here' in non-quotation with a non-shifted prime, models score below 2% because they follow the context rather than the grammar. Direct quotation reverses some of these patterns: it improves 'here' accuracy dramatically (up to above 97% for the best model) while reducing accuracy for 'you' and 'tomorrow', and 'tomorrow' shows a blanket bias toward non-shifted readings regardless of condition. The paper concludes that LLMs do not uniformly apply the semantic rule that non-quoted indexicals refer to the actual utterance context and quoted indexicals shift to the reported context; instead, they treat different indexicals with different strategies.

Load-bearing premise

The benchmark's ground truth is trusted for all 1,600 items even though only 400 were human-checked; if the remaining 1,200 items contain systematic label errors, the reported accuracy gaps, including the near-perfect 'I' scores, could be artifacts.

Editorial extensions

If this is right

  • Models can reach near-ceiling accuracy on 'I' while failing on 'you', 'here', and 'tomorrow', so benchmarks that test only third-person pronouns cannot predict indexical competence.
  • In non-quoted sentences, context primes can override the grammatically correct reading for 'you' and 'here', meaning pragmatically irrelevant narrative cues are a source of error.
  • Direct quotation is not a uniformly helpful cue: it improves 'here' accuracy, reduces 'you' and 'tomorrow' accuracy, and leaves 'tomorrow' biased toward non-shifted readings.
  • The strong non-shifted bias for 'tomorrow' creates an illusion of high accuracy in non-quotation conditions while producing near-zero accuracy in quotation conditions, so headline numbers must be read per-condition.
  • Because the results diverge from the earlier Turkish 'ben/I' finding, the same benchmark design can expose language-specific indexical behavior, such as differences in pro-drop languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the context-prime effect is robust, LLM coreference evaluations should include deictic items, since passing narrative pronoun tests may overstate real conversational competence.
  • The opposite sign of quotation for 'here' versus 'you' and 'tomorrow' suggests models may treat quotation as a format or lexical cue rather than as a semantic shift operator; a testable prediction is that changing only the punctuation while keeping wording identical will flip accuracy for 'here' but not for 'you'.
  • The near-ceiling performance on 'I' may reflect first-person frequency in training text; a testable extension is probing with rare or invented names, or with pro-drop constructions, to see whether 'I' accuracy degrades.
  • Forced-choice evaluation may amplify response biases; an open-ended generation or confidence-rated probe could reveal whether models know the correct reading but fail under constraint.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces the English Indexical Dataset, a set of 1,600 multiple-choice items probing how large language models interpret the indexicals 'I', 'you', 'here', and 'tomorrow' under shifted versus non-shifted context primes and in quoted versus non-quoted sentences. The authors evaluate four proprietary LLMs (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, DeepSeek-V3) and report that accuracy is not uniform across indexical types: models perform near-optimally on 'I', poorly on 'you', 'here', and 'tomorrow', and quotation improves 'here' accuracy while lowering 'you' and 'tomorrow' accuracy. The paper also releases code and data.

Significance. If the reported findings are sound, this would be the first English-language benchmark for LLM interpretation of indexicals and a useful diagnostic for context-sensitive coreference. The linguistic motivation is clear and the dataset is a potentially valuable community resource, particularly because the authors ship code and data. However, the validity of the headline results is currently undermined by an internal inconsistency in the sample tables (reversed option labels for 'you' and 'tomorrow') and by limited manual validation of the dataset, so the significance is conditional until these issues are resolved.

major comments (3)
  1. [Appendix A.2, Tables 10 and 12] This is the load-bearing issue.
  2. [Section 3.1] Only 25% of the dataset (400 trials) was manually validated; the remaining 1,200 items were accepted as generated without human checking. Since GPT-4o generated the items and is also one of the evaluated models, systematic label errors in the unchecked portion could affect the per-indexical accuracies, including the near-ceiling 'I' scores. The authors should either provide item-level validation for the full dataset, report human-annotator agreement statistics, or explicitly report error bounds that account for unverified labels.
  3. [Section 3.3 and Figure 2] The evaluation uses a fixed two-option format with a single option order and reports point estimates without confidence intervals or repeated runs. If the 'shifted' option is always listed first or second, an order bias could masquerade as a linguistic effect; the differences that drive the claims (e.g., quotation affecting 'you' versus 'here') may be within sampling noise. The paper should report exact binomial confidence intervals for the reported accuracies or, ideally, run the models multiple times and report means and variances.
minor comments (6)
  1. [Section 1] The word 'interpreteting' should be 'interpreting'.
  2. [Section 3.1] 'Compromising' should be 'comprising' in the sentence describing the dataset size.
  3. [Figure 2 caption] The caption reads 'Performance analysis plot of for the indexical'; it should be 'Performance analysis plot for the indexical'.
  4. [Appendix A.2] The sample items are highly formulaic and appear to follow the exact template from the generation prompts; the paper should clarify whether the final dataset items all share this structure, as this may limit the generalizability of the conclusions.
  5. [Section 7 Limitations] The limitations section only acknowledges the black-box setting; it should also mention the unvalidated 75% of items and the fixed answer-order design as limitations.
  6. [References] Some references are incomplete, for example 'Anthropic. The claude 3 model family: Opus, sonnet, haiku.' should include the year and a fuller bibliographic entry.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper measures LLM accuracy against an external linguistic ground truth; the main risks are data-labeling inconsistencies in Appendix Tables 10 and 12 and the GPT-4o-as-generator overlap, not circular derivation.

full rationale

The paper's central claims are direct accuracy measurements against a fixed ground-truth key defined by standard indexical semantics (Kaplan 1977; Schlenker 2003), not by model outputs. The evaluation metric in Section 3.4 sets the optimal policy from the linguistic rule that quotation shifts indexicals and non-quotation does not; this is an external criterion, not fitted to model behavior. No parameter is fit, and no equation reduces a predicted quantity to an input. The self-citations (Oğuz et al. 2024, Oğuz et al. 2020) are used only to contextualize or contrast results, not to prove the English claims. The use of GPT-4o to generate items while also evaluating GPT-4o is a benchmark-validity caveat, but it does not make the accuracy scores equivalent to the item-generation process by construction; the ground truth is assigned from the experimental condition, not from GPT-4o's judgments. Appendix Tables 10 and 12 contain an apparent inversion of the 'Shifted' and 'Non-shifted' option labels for 'you' and 'tomorrow' (e.g., the actual addressee Ryan is listed as the shifted option and Dylan's addressee Samuel as the non-shifted option), which could undermine those specific accuracy numbers if the released dataset's answer-key fields mirror the tables; but this is a labeling/quality concern, not a circular derivation. Given these considerations, the derivation chain is self-contained and no circular step is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central evaluation rests on the semantic theory of indexicality, the assumption that unchecked GPT-4o-generated items are correctly labeled, and the validity of the forced-choice probe. No fitted free parameters and no invented entities are involved.

assumptions (3)
  • domain assumption Indexicals (I, you, here, tomorrow) unambiguously refer to actual speech-act coordinates; direct quotation shifts them to the reported context.
    Adopted from Kaplan (1977) and Schlenker (2003) in Section 2; the benchmark ground truth is built on this semantic theory.
  • ad hoc to paper The 400 human-checked trials are representative, and the remaining 1,200 GPT-4o-generated trials are correctly labeled and grammatically valid.
    Section 3.1 states only 25% of the dataset was verified; the validity of the rest is assumed.
  • ad hoc to paper The forced two-option question format measures indexical interpretation rather than option-order or wording artifacts.
    Appendix B.1 shows a fixed prompt with no counterbalancing mentioned; the interpretation hinges on this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Un-considering Contextual Information: Assessing LLMs' Understanding of Indexical Elements." pith.science (2026). https://pith.science/paper/DLZITXZ7

@misc{pith2026250601089,
  author       = {Pith},
  title        = {Pith review of: Un-considering Contextual Information: Assessing LLMs' Understanding of Indexical Elements},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DLZITXZ7}},
  note         = {Machine review of arXiv:2506.01089}
}
read the original abstract

Large Language Models (LLMs) have demonstrated impressive performances in tasks related to coreference resolution. However, previous studies mostly assessed LLM performance on coreference resolution with nouns and third person pronouns. This study evaluates LLM performance on coreference resolution with indexical like I, you, here and tomorrow, which come with unique challenges due to their linguistic properties. We present the first study examining how LLMs interpret indexicals in English, releasing the English Indexical Dataset with 1600 multiple-choice questions. We evaluate pioneering LLMs, including GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and DeepSeek V3. Our results reveal that LLMs exhibit an impressive performance with some indexicals (I), while struggling with others (you, here, tomorrow), and that syntactic cues (e.g. quotation) contribute to LLM performance with some indexicals, while they reduce performance with others. Code and data are available at: https://github.com/metehanoguzz/LLMs-Indexicals-English.

Figures

Figures reproduced from arXiv: 2506.01089 by the authors.

Figure 1
Figure 1. An example for LLM misinterpreting indexical element ‘here’, uttered by a speaker in Los Angeles. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. From left to right: Performance analysis plot of for the indexical ‘I’, Performance analysis plot of for the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 14 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David A. Sontag. 2022. Large language models are few-shot clinical information extractors. In Conference on Empirical Methods in Natural Language Processing

    work page 2022

  4. [4]

    https://api.semanticscholar.org/CorpusID:268232499 The claude 3 model family: Opus, sonnet, haiku

    Anthropic. https://api.semanticscholar.org/CorpusID:268232499 The claude 3 model family: Opus, sonnet, haiku

    work page

  5. [5]

    Yavuz Faruk Bakman, Duygu Nur Yaldiz, Baturalp Buyukates, Chenyang Tao, Dimitrios Dimitriadis, and Salman Avestimehr. 2024. https://doi.org/10.18653/v1/2024.acl-long.419 MARS : Meaning-aware response scoring for uncertainty estimation in generative LLM s . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1...

    doi:10.18653/v1/2024.acl-long.419 2024

  6. [6]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, S...

    arXiv 2020

  7. [7]

    Crawley, Rosemary J

    Rosalind A. Crawley, Rosemary J. Stevenson, and David Kleinman. 1990. https://doi.org/10.1007/BF01077259 The use of heuristic strategies in the interpretation of pronouns . Journal of Psycholinguistic Research, 14

    work page doi:10.1007/bf01077259 1990

  8. [8]

    Amy Rose Deal. 2020. A theory of indexical shift: meaning, grammar, and crosslinguistic variation. MIT Press, Boston, MA

    work page 2020

Show all 23 references
  1. [9]

    DeepSeek-AI. 2024. https://arxiv.org/abs/2412.19437 Deepseek-v3 technical report . Preprint, arXiv:2412.19437

    2024 arXiv

  2. [10]

    Yujian Gan, Massimo Poesio, and Juntao Yu. 2024. Assessing the capabilities of large language models in coreference: An evaluation. In International Conference on Language Resources and Evaluation

    2024

  3. [11]

    David Kaplan. 1977. Demonstratives: An essay on the semantics, logic, metaphysics, and epistemology of demonstratives and other indexicals. Themes from Kaplan, pages 565--614

    1977

  4. [12]

    Le and Alan Ritter

    Nghia T. Le and Alan Ritter. 2023. Are large language models robust coreference resolvers?

    2023

  5. [13]

    Metehan O g uz, Yusuf Ciftci, and Yavuz Faruk Bakman. 2024. https://aclanthology.org/2024.sigturk-1.5 Do LLM s recognize me, when I is not me: Assessment of LLM s understanding of T urkish indexical pronouns in indexical shift contexts . In Proceedings of the First Workshop on...

    2024

  6. [14]

    OpenAI. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  7. [15]

    Metehan Oğuz, Burak Öney, and Dennis Ryan Storoshenko. 2020. Obligatory indexical shift in Turkish . In Proceedings of Canadian Linguistic Association (CLA), Western University, London, ON, Canada

    2020

  8. [16]

    Martin Pickering and Asifa Majid. 2007. What are implicit causality and consequentiality? Language & Cognitive Processes, 22

    2007

  9. [17]

    Philippe Schlenker. 2003. A plea for monsters. Linguistics and Philosophy, 26:29--120

    2003

  10. [18]

    Stewart and Martin Pickering

    Andrew J. Stewart and Martin Pickering. 1998. Implicit consequentiality. In Proceedings of the 20th Annual Conference of the Cognitive Science Society

    1998

  11. [19]

    Gemini Team. 2024. https://arxiv.org/abs/2403.05530 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context . Preprint, arXiv:2403.05530

    2024 arXiv

  12. [20]

    Duygu Nur Yaldiz, Yavuz Faruk Bakman, Baturalp Buyukates, Chenyang Tao, Anil Ramakrishna, Dimitrios Dimitriadis, Jieyu Zhao, and Salman Avestimehr. 2024. https://arxiv.org/abs/2406.11278 Do not design, learn: A trainable scoring function for uncertainty estimation in generativ...

    2024 arXiv

  13. [21]

    Xiaohan Yang, Eduardo Peynetti, Vasco Meerman, and Christy Tanner. 2022. https://api.semanticscholar.org/CorpusID:248780307 What gpt knows about who is who . In First Workshop on Insights from Negative Results in NLP

    2022

  14. [22]

    Xiulin Yang. 2025. https://aclanthology.org/2025.coling-main.257/ Language models at the syntax-semantics interface: A case study of the long-distance binding of C hinese reflexive ziji . In Proceedings of the 31st International Conference on Computational Linguistics, pages 3...

    2025

  15. [23]

    Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, Jie Zhou, Siming Chen, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. https://arxiv.org/abs/2303.10420 A comprehensive capability analysis of GPT-3 and GPT-3.5 series...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.