Pith. sign in

REVIEW 4 major objections 4 minor 8 references

Three LLM agents, given 100 chances each, never completed a single conversation in a newly constructed language, though their errors resembled human babbling and combinatorial testing.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

LLM agents (GPT-4o-mini, Gemini-2.5-flash, Claude-3.5-haiku) cannot learn an enumerated synthetic language through feedback within 100 turns, despite the task's grammar being fully specified in the system prompt.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection An interactive evaluation protocol worth borrowing, but its synthetic 'language' is a 100-entry lookup table — so the failure result is about memorization, not acquisition. the 4 major comments →

arxiv 2509.07389 v1 pith:FXIEE7BA submitted 2025-09-09 cs.CL cs.AIcs.HCcs.LG

Talking with Oompa Loompas: A novel framework for evaluating linguistic acquisition of LLM agents

Sankalp Tattwadarshi Swain , Anshika Krishnatray , Dhruv Kumar , Jagat Sesh Challa This is my paper
classification cs.CL cs.AIcs.HCcs.LG
keywords constructed language acquisitionLLM agentsinteractive feedbackTinkatongueevaluation benchmarkbabbling and imitation strategiesturn validityfeedback responsiveness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that testing whether an LLM can acquire a language requires dropping it into a live conversation with a bot that speaks only a made-up language and gives yes/no feedback, with no pretraining exposure. It builds such a test around a 100-sentence language called Tinkatongue and runs three agents for up to 100 responses each. Across ten trials per agent, no model completed the required four-turn conversation, and turn-validity rates were low for all; one model adapted faster, but every model recovered from negative feedback. The authors also read the transcripts as showing babbling, imitation, and systematic combinatorial testing, strategies that mirror early human language learning. If true, the result shifts evaluation from static benchmarks toward interactive, feedback-driven acquisition and suggests that current agents can adapt locally but cannot sustain a novel conversation.

Core claim

The paper's claim is that LLM agents, given no examples of Tinkatongue, cannot bootstrap enough of the language from interaction alone to carry a four-turn conversation: zero of the sessions produced the three successful conversations required, and no model completed even one conversation within 100 responses. The authors interpret this not as pure failure but as evidence of stage-like adaptation: agents produced isolated valid utterances (Claude-3.5-haiku in 8 of 10 trials, Gemini in 6, GPT-4o-mini in 5), immediately reoriented after the negative feedback 'moko lira bani', yet never internalized adjacency constraints, so valid turns did not chain. The qualitative transcript shows systematic

What carries the argument

Tinkatongue, a finite constructed language of 100 enumerated three-word sentences and 25 four-turn conversations linked by word overlap, with membership decided by table lookup rather than generated by rules. The deterministic Oompa Loompa bot is the second piece: it replies 'koro + next sentence' to valid continuations and 'moko lira bani' to invalid ones. Together they turn 'acquire a language' into a measurable reinforcement problem, and the four custom metrics (turn validity, feedback responsiveness, adjacency compliance, time to first positive feedback) are how the paper claims to observe adaptation.

Load-bearing premise

The load-bearing assumption is that learning to reproduce 100 pre-enumerated valid sentences under binary feedback counts as language acquisition; because validity is defined by table lookup rather than any generative rule, the failure results describe lookup-table learning unless that equivalence is accepted.

What would settle it

Recompute Feedback Responsiveness from the Appendix E transcript instead of the reported aggregates: whenever an invalid reply immediately follows 'moko lira bani', it should count as a failed recovery, and the transcript shows several such cases; if the recomputed FR is below 1.0, the reported metric is wrong and the 'perfect recovery' conclusion does not follow.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Static benchmarks can now be supplemented with a runtime acquisition score, measuring whether a model can learn from its own errors rather than from pretraining exposure.
  • The zero-completion result predicts that current off-the-shelf LLM agents cannot be dropped into a human-style immersion setting and sustain a novel-language exchange, even when the language is tiny and fully enumerable.
  • Perfect feedback responsiveness across models suggests that negative feedback does not need to be more elaborate to trigger immediate recovery; the bottleneck is conversation-level adjacency, not correction itself.
  • The observed strategy differences (babbling vs imitation vs combinatorial search) give a taxonomy for classifying agent exploration behavior in interactive learning tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The formal language has no generative grammar: the 100 valid sentences are merely enumerated, so 'acquiring' Tinkatongue amounts to memorizing a lookup table from feedback. A reader who equates language acquisition with rule induction should not take the zero-completion result as evidence about generalization.
  • Replacing the enumerated set with a small productive grammar would be a natural, harder version of the same test; if agents then generated novel valid sentences, it would show the bottleneck is rule discovery, not interaction itself.
  • The strategy labels (babbling, imitation, combinatorial testing) could be turned into automatic behavioral classifiers, letting future work measure exploration style as a model property rather than a transcript impression.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes an interactive evaluation framework in which an LLM agent must learn a constructed language, Tinkatongue, by conversing with a deterministic bot under binary feedback. The agent receives only the bot's utterances and 'koro'/'moko lira bani' signals. Three commercial LLMs (GPT-4o-mini, Gemini-2.5-flash, Claude-3.5-haiku) are run for ten trials each with a 100-response budget. The central reported findings are: (i) no model completes a conversation within 100 responses; (ii) Claude-3.5-haiku has substantially higher Turn Validity Rate than the other two; (iii) all models have perfect Feedback Responsiveness (FR = 1.0); and (iv) qualitative transcript analysis shows strategies 'mirroring' human babbling, imitation, and combinatorial testing. The paper concludes that the framework is a promising benchmark for studying feedback-driven language acquisition in LLM agents.

Significance. If the central claim were supported, the paper would introduce a useful benchmark for interactive language acquisition: controlled artificial languages, deterministic feedback, and reproducible metrics would be valuable. The paper also has positive features: the protocol is explicit, the formal specification is given in an appendix, and the raw transcript in Appendix E is a useful artifact for checking claims. However, the operationalization of 'language acquisition' is not valid as stated. Tinkatongue is defined as an arbitrary enumerated set of 100 sentences (Appendix A.2), with validity checked by lookup (Eq. 6), and the paper itself states there is 'no provision for novel sentence generation.' The task is therefore finite-set memorization under feedback, not acquisition of a generative linguistic system. In addition, the perfect FR scores in Table 1 are contradicted by the Appendix E transcript, where models emit long runs of invalid sentences after negative feedback. These two problems are load-bearing: the advertised conclusions about linguistic acquisition and about feedback responsiveness do not follow from the reported data. With the current design, the paper cannot suppor

major comments (4)
  1. [§3, Appendix A.2, Eqs. (5)–(6)] The formal language Lsent is defined by enumeration of 100 triples, and membership is checked by table lookup. The paper explicitly states in §3: 'The language is exhaustive and contains 25 predefined conversations... with no provision for novel sentence generation.' There is no generative grammar or productive rule connecting the valid sentences; the constraints given (bisyllabic words, 3-word sentences, shared-word adjacency) are necessary but not sufficient. The agent's task is therefore to memorize 100 arbitrary triples under binary feedback, not to acquire a language. The failure to discover the finite lookup table says little about 'linguistic acquisition through pattern recognition,' and the claim in the abstract that models 'fail to establish a conversation' as evidence of failed language acquisition is not supported.
  2. [§3, Eq. (2), Table 1 vs. Appendix E, Table 6] Feedback Responsiveness is defined as feedback_recoveries / feedback_opportunities, and the text describes an 'immediately subsequent valid reply' as an 'immediate recovery.' Table 1 reports FR = 1.0 for every model in every trial (zero variance). This is contradicted by the trial transcript in Appendix E. For example, after the first 'moko lira bani' in Table 6, the model replies 'tira lomo banu' (invalid), then 'kina sora lomo' (invalid), and continues through many invalid responses before producing a valid one. If 'feedback_recoveries' means immediate recovery, those trials should have FR values well below 1. If it means any later recovery, the definition is not what is described and the metric is not interpretable. The reported perfect FR cannot be reproduced from the paper's own transcript, so the claim of 'perfect responsiveness to negative feedback' is unreliable.
  3. [§2, Appendix C (system prompt)] The system prompt explicitly informs the model of the syntactic constraints: 'Each word is bisyllabic, each sentence consisting of three words,' and 'Every consecutive sentence in a conversation has at least one common word.' This removes the need for syntactic pattern recognition during interaction; the agent is told the grammar. The only unknown is which of the combinatorially possible triples appear in the predefined list, and the prompt also says 'All conversations are predefined.' The experiment therefore tests lookup-table search under feedback, not acquisition of syntax or word order from exposure. A genuine language-acquisition evaluation would need a generative grammar and would not disclose the full rule set in the prompt.
  4. [§3 (Results, qualitative analysis)] The claim that models 'adopt distinct strategies that mirror human approaches to language learning' (abstract and §3) is based on informal transcript inspection (Table 2 and Appendix E). No coding scheme, inter-annotator agreement, hypothesis-testing procedure, or comparison baseline is provided. The labels 'babbling,' 'imitation,' and 'combinatorial testing' are imposed post hoc on transcripts of a finite memorization task. These qualitative interpretations are anecdotal and cannot support the strong claim about human-like acquisition strategies. At minimum, a systematic content analysis with defined categories and reliability statistics would be required.
minor comments (4)
  1. [Throughout] There are several typos and small inconsistencies: 'agentS' in the introduction, 'acqusition' at the end of §3, and 'Tmax' in §2 is not defined or reported. The caption for Table 2 says 'Transcript Excerpt for Gemini-2.5-flash' but the text refers to the condition with a modified system prompt; please clarify which condition is shown.
  2. [§3, Table 1] The table is dense and does not report the primary outcome of interest: the number of completed conversations (or 'successful conversations' per Eq. (13)). The reader cannot verify the claim that 'no model achieved a fully successful conversation within 100 responses' from the reported metrics alone. A dedicated column for conversation completion would be necessary.
  3. [Appendix A.2, Eq. (7)] The adjacency relation is defined as a non-empty set intersection over words, but words within a sentence are a multiset (e.g., 'soro kina soro'). Please clarify whether duplicate words are treated as set elements or multiset elements; the current notation '{w ∈ u}' is ambiguous.
  4. [Appendix E, Table 6] The transcript shows 'Valid (count: 25)' at the end, but it is not clear whether these are counted valid sentences or something else. The heading says 'Validity' but the count column is unexplained. Please define the count column explicitly.

Circularity Check

1 steps flagged

Quantitative evaluation is self-contained; only the qualitative human-mirroring interpretation is a post-hoc label restated as evidence.

specific steps
  1. renaming known result [Section 3, 'Results/Qualitative analysis' and Appendix E Table 6 (transcript of Gemini-2.5-flash)]
    "The analysis reveals that LLM agents adopt learning strategies that closely parallel early stages of human language acquisition. ... Gemini-2.5-flash babbled, resembling a baby trying to learn words."

    The only evidence for the conclusion that the agents 'mirror human approaches' is the post-hoc naming of the models' repetitive/invalid outputs as 'babbling' and 'imitation.' No independent coding scheme, pre-registered criteria, or human-babbling comparison is provided, so the claimed discovery ('models use human-like strategies') is the same as the label imposed on the transcripts. The quantitative failure result does not depend on this labeling, so the circularity is confined to the qualitative interpretation.

full rationale

The paper's formal chain is not circular: Tinkatongue is defined by enumeration (Eq. 5-6), conversation success by matching predefined sequences (Eq. 8, 13), and the reported metrics (TVR, AC, FR, TTFK) are direct counts over transcripts. No parameter is fitted, no prediction is derived from a fitted value, and the references are all external (GLUE, SuperGLUE, MMLU, etc.), with no self-citation chain carrying the argument. The central claim that no model completed a conversation within 100 responses is a direct measurement of the transcripts. The weakness flagged by the skeptical reading — that the enumerated lookup-table language makes the result about memorization rather than linguistic acquisition — is a construct-validity concern, not a circular reduction: the conclusion does not follow from the definitions alone; it depends on an auxiliary assumption about what counts as language acquisition. The only genuinely circular element is qualitative: the 'strategies that mirror human language acquisition' (babbling, imitation) are post-hoc labels applied to model outputs; the conclusion then restates the label as evidence. I also note (as a correctness issue, not circularity) that the reported perfect FR=1.0 is difficult to reconcile with Appendix E, where invalid responses follow 'moko lira bani' repeatedly. That inconsistency does not affect the circularity score.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 1 invented entities

The ledger captures the paper's core assumptions. There are no fitted parameters; the main burden is the conceptual leap from an enumerated lookup table to 'language acquisition'. The invented entity is the test language itself, which the authors treat as a language despite having no generative structure.

axioms (4)
  • ad hoc to paper Tinkatongue's 100-sentence enumeration is a legitimate operationalization of 'a language to be acquired by pattern recognition.'
    The validity predicate (Eq. 6) is membership by enumeration, so there is no generative pattern to infer. The paper's framing of the task as language acquisition assumes this without argument.
  • domain assumption The system prompt's grammar hints do not give away the answer set.
    Appendix C tells the agent that sentences have three bisyllabic words and share a word with the previous sentence, but not which triples are in the 100-sentence list. The paper assumes the remaining task is learnable from feedback rather than pure memorization.
  • domain assumption The binary feedback tokens are unambiguous to the models.
    The feedback policy (Eq. 9) maps valid to 'koro' and invalid to 'moko lira bani'; the paper assumes all models can parse this as success/failure and use it to improve.
  • ad hoc to paper Qualitative transcript categories correspond to human acquisition stages.
    Section 3 claims 'strategies that closely parallel early stages of human language acquisition' based on informal inspection, without a human comparison group or a falsifiable coding scheme.
invented entities (1)
  • Tinkatongue (constructed language) no independent evidence
    purpose: The synthetic language used to test acquisition; a fixed list of 100 valid triples and 25 conversations.
    Not a physical or explanatory postulate; it is a dataset constructed by the authors. It has no falsifiable handle outside this experiment, and its validity as a proxy for human language acquisition is the paper's central, unsupported assumption.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Talking with Oompa Loompas: A novel framework for evaluating linguistic acquisition of LLM agents." pith.science (2026). https://pith.science/paper/FXIEE7BA

@misc{pith2026250907389,
  author       = {Pith},
  title        = {Pith review of: Talking with Oompa Loompas: A novel framework for evaluating linguistic acquisition of LLM agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FXIEE7BA}},
  note         = {Machine review of arXiv:2509.07389}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Existing evaluation studies on linguistic competence of large language models (LLM agents) have focused primarily on vocabulary learning, morphological rule induction, syntactic generalization, pragmatic inference, and cross-linguistic transfer. However, none assess whether LLM agents can acquire a language through pattern recognition and interactive feedback, a central feature of human language acquisition. We propose a novel experimental framework in which an LLM agent is evaluated on its ability to acquire and use a newly constructed language (Tinkatongue) in conversation with a bot that understands only Tinkatongue. Our findings show that LLM agents fail to establish a conversation within 100 responses, yet they adopt distinct strategies that mirror human approaches to language learning. The results suggest a new direction for evaluation benchmarks and open pathways to model designs that learn more effectively from interactive feedback.

Figures

Figures reproduced from arXiv: 2509.07389 by Anshika Krishnatray, Dhruv Kumar, Jagat Sesh Challa, Sankalp Tattwadarshi Swain.

Figure 1
Figure 1. Figure 1: Side-by-side comparison: (a), (b) - conversation outcomes; (c) reply flow. Note: The white chat boxes [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

8 extracted references · 6 canonical work pages · 4 internal anchors

  1. [1]

    On the cross-lingual transferability of monolingual representations

    Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolingual representations. In Annual Meeting of the Association for Computational Linguistics, 2019. URL https://api.semanticscholar.org/CorpusID:204901567

    work page 2019

  2. [2]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv, abs/2009.03300, 2020. URL https://api.semanticscholar.org/CorpusID:221516475

    Pith/arXiv arXiv 2009

  3. [3]

    Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger P. Levy. A systematic assessment of syntactic generalization in neural language models, 2020. URL https://arxiv.org/abs/2005.03692

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  4. [4]

    Novel-WD: Exploring acquisition of Novel World Knowledge in LLMs Using Prefix-Tuning

    Maxime M'eloux and Christophe Cerisara. Novel-wd: Exploring acquisition of novel world knowledge in llms using prefix-tuning. ArXiv, abs/2408.17070, 2024. URL https://api.semanticscholar.org/CorpusID:271893019

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  5. [5]

    MultiPragEval: Multilingual Pragmatic Evaluation of Large Language Models

    Dojun Park, Jiwoo Lee, Seohyun Park, Hyeyun Jeong, Youngeun Koo, Soonha Hwang, Seonwoo Park, and Sungeun Lee. Multiprageval: Multilingual pragmatic evaluation of large language models. ArXiv, abs/2406.07736, 2024. URL https://api.semanticscholar.org/CorpusID:270392017

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  6. [6]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In BlackboxNLP@EMNLP, 2018. URL https://api.semanticscholar.org/CorpusID:5034059

    work page 2018

  7. [7]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. ArXiv, abs/1905.00537, 2019. URL https://api.semanticscholar.org/CorpusID:143424870

    Pith/arXiv arXiv 1905

  8. [8]

    Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model

    Leonie Weissweiler, Valentin Hofmann, Anjali Kantharuban, Anna Cai, Ritam Dutt, Amey Hengle, Anubha Kabra, Atharva Kulkarni, Abhishek Vijayakumar, Haofei Yu, Hinrich Sch \"u tze, Kemal Oflazer, and David R. Mortensen. Counting the bugs in chatgpt's wugs: A multilingual investigation into the morphological capabilities of a large language model. ArXiv, abs...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.