Pith. sign in

REVIEW 4 minor 45 references

Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research

T0 review · 0 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read A new NLP task or dataset is only progress if it states which language capabilities it exercises; otherwise, the paper argues, the contribution is vacuous.

desk verdict A genuinely useful methodological critique that gives NLP researchers a cleaner vocabulary for task design, and the formal appendix does real work. read the letter →

arxiv 1908.10747 v1 pith:CA6R4A4V submitted 2019-08-28 cs.CL cs.AI

classification cs.CLcs.AI
keywords languagetasksgamesNLPmethodologybenchmarkevaluationcompetencedatasetvalidationcapabilitydecompositionintelligentagentenvironments
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper examines the common NLP practice of contributing a new task and dataset and asks why such contributions are assumed to be progress. It proposes precise definitions of a language task, a micro-world or environment, and a language game, and it argues that a new task or dataset counts as progress toward modeling general language competence only if its relation to the capabilities of a competent language user is made explicit. The author's central point is that motivating a task in itself requires claims about which capability subset the task involves; how good such motivation is depends on how precisely and falsifiably those claims are stated. If the paper is right, benchmark introductions should come with explicit capability analyses, and dataset deficiencies can be detected by showing that a model solves the task without information the task description treats as essential.

What carries the argument

The machinery is a set of formal tuple definitions — language task $(S, A, L, D_T)$, micro-world/environment $(S, A, E, R, D_W)$, interaction game $(P, A, o, T, E, D_G)$ with a disinterested player Nature — together with the 'involves' relation between a task and a capability subset. The definitions give a language to compare tasks, datasets, environments, and games, and the 'involves' relation supplies the yardstick: a contribution is progressive only when the claimed capability subset is specified precisely enough to be tested and is related to the overall capability set of a competent language user.

What would settle it

Run a large battery of diverse language tasks with human participants and factor-analyze the performance matrix. If no stable task-specific capability components emerge — that is, a single general language factor explains nearly all variance — then the paper's assumption that the overall capability set CL decomposes into task-linked subsets CT loses its empirical grounding.

Watch

Extended reading notes

Core claim

The central claim is that a new NLP task or dataset only constitutes progress if the relation between the task and the capabilities of a competent language user is stated explicitly; otherwise, the contribution is vacuous. The paper grounds this claim by defining a language task as a mapping from states to actions, at least one side involving natural language expressions, conforming to a task description; an environment as an action-to-state mapping conforming to a world description; and an interaction game as players, action spaces, an observability function, a turn-taking rule, an evaluation rule, and a game description. It then argues that any task motivation presupposes a set CL of capabilities of a competent language user and a subset CT that the task exercises. The strength of the motivation is ranked by how precisely CT is specified and by two dimensions: separability (whether CT can be handled independently of the rest of CL) and exhaustivity (whether the task exercises all of CT). A claim like 'the task involves the capability to compute syntactic structure' is strong because it could be wrong; a claim like 'the task involves the capability to do the task' is trivial.

Load-bearing premise

The argument depends on there being a well-defined set of capabilities of a competent language user that can be broken into subsets tied to individual tasks; if that decomposition does not exist, evaluating tasks by their claimed capability links has no foundation.

Editorial extensions

If this is right

  • If the paper is right, a new task proposal should state, in falsifiable terms, which subset of competent-language-user capabilities the task engages, and how that subset relates to subsets of existing tasks.
  • A dataset can be rejected as an unsatisfactory exemplification of a task when a model solves it without information the task description deems crucial, as happened when visual question answering was solved without visual input.
  • Progress in tasks themselves means arguing that the new task's capability subset is more specific, more separable, or more exhaustive than the old one, not merely that a new dataset has been collected.
  • Language games can be assessed by how close they come to unrestricted situated language interaction, the natural upper bound for language-game design.
  • Probing results on trained models (for example, finding syntactic structure inside a network) should be read as evidence about whether a postulated capability is needed for the task, rather than as standalone architectural curiosities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the author leaves implicit is a 'capability sheet' for each dataset or task, listing the claimed CT, its separability and exhaustivity, and the evidence that would falsify the claim; this would make the 'involves' relation contestable by inspection.
  • If the capability-decomposition view is correct, then transfer learning provides a direct test: a model trained on a task that genuinely engages CT should improve performance on another task engaging the same CT; failures of transfer would indicate the capability claim was wrong.
  • The framework could be applied to evaluate 'AI-complete' claims: saying a task is AI-complete amounts to claiming CT = CL, and by the paper's ranking that is the strongest and least likely kind of capability claim, not a casual rhetorical flourish.
  • Operationalizing 'closeness to unrestricted situated language interaction' could yield a graded benchmark-construction guide (e.g., increasing the number of interacting players, hiding observations, or lengthening action sequences), although the paper does not specify such a metric.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 4 minor

Summary. This position paper argues that current NLP research habitually introduces new tasks, datasets, worlds, and games while leaving implicit what such contributions are progress toward. The author defines a language task as a mapping between input and output/action spaces involving natural language, a micro-world/environment as a mapping from actions to environmental responses, and an interaction game as a setting with repeated, connected language tasks regulated by turn-taking and evaluation rules. The central normative claim is that motivating a task or game requires making explicit claims about how the capabilities C_T it involves relate to the larger set C_L of capabilities of a competent language user; without such claims, a new task or dataset does not by itself constitute progress. The paper distinguishes three modes of progress (better models, better datasets, better tasks/games), analyzes their different argumentative burdens, and recommends a stronger connection to linguistics and cognitive psychology to make capability claims contestable. Formal definitions are provided in an appendix.

Significance. If the argument is accepted, it gives researchers a vocabulary for evaluating task and dataset contributions beyond benchmark scores, and it sharpens a critique that is often voiced only informally. The paper's main strength is that it makes an implicit argument structure explicit: progress claims rest on assumptions about the decomposition of language competence, and those assumptions should be stated so they can be challenged. The treatment of the VQA language-bias example and the discussion of dataset validity are concrete and instructive. The paper also honestly flags its own limitations, most notably in Section 3.1, where it acknowledges that motivating a task requires assumptions about how C_L decomposes. The paper contains no technical derivations or experiments, so its contribution is conceptual and methodological; it is a well-targeted piece of scientific criticism rather than a new empirical result. The central claim is conditional and remains coherent even without a fully specified theory of language capabilities.

minor comments (4)
  1. [Section 3.1] The ranking of capability claims from the trivial "task T involves the capability to do task T" to stronger falsifiable claims is intuitive, but the "involves" relation itself is never defined formally; a short worked example showing how a specific task would be paired with a non-trivial, falsifiable capability claim would make the central recommendation easier to apply.
  2. [Section 2.3 and Section 4.2] There are several typographical errors: "priviledged" in Section 2.3 should be "privileged", "Targetting" in Section 4.2 should be "Targeting", and "diagramm" in Section 4.3 should be "diagram".
  3. [Appendix A.1, Definition 1] The condition "with either the states in S or the actions in A (or both) having as part natural language expressions" is grammatically awkward and could be misread as requiring states or actions themselves to contain language expressions; a cleaner formulation would describe a function from states or actions to natural language expressions.
  4. [Section 4.3] The statement that tasks "are only grounded (to their left in the diagram) by capabilities" relies on the spatial layout of Figure 2; consider stating the direction of grounding textually without reference to the diagram's left side, since the figure may be rendered or read differently.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity identified: the paper is a methodological/normative framework with no fitted inputs or self-citation load-bearing derivation.

full rationale

No circularity found. The paper makes no empirical prediction and fits no parameters; its central claim is conditional and normative: contributions to NLP only count as progress toward modelling general language competence if their relation to a set C_L of capabilities is made explicit. The formal definitions in Appendix A are stipulative and do not presuppose the conclusion. The argument in Section 3.1 explicitly ranks capability claims from trivial ('task T involves the capability to do task T') to falsifiable strong claims, and concedes that motivating tasks requires assumptions about how C_L decomposes; this is an acknowledged limitation, not a circular step. The only self-citation (Ilinykh et al. 2018) is used as an external empirical pointer to production differences, not as the justification for the paper's framework. Hence the derivation chain is self-contained: no equation reduces to its input, and no prediction is renamed as a fit.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

No numerical fits or new physical entities. The paper introduces formal definitions (task, world, game) that are conceptual tools; the load-bearing assumptions concern the existence and decomposability of language capabilities.

assumptions (2)
  • domain assumption There exists a set CL of capabilities of a competent language user.
    Introduced in Section 3.1; the paper treats CL as a well-defined given that tasks can be related to.
  • domain assumption For any task T, the set of required capabilities CT is a subset of CL, and CT can be characterized separately from the rest of CL.
    Section 3.1 states 'To be good at task T, an agent must possess a set CT of capabilities... CT ⊆ CL'. This separability is the crux of the evaluation scheme.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research." pith.science (2026). https://pith.science/paper/CA6R4A4V

@misc{pith2026190810747,
  author       = {Pith},
  title        = {Pith review of: Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CA6R4A4V}},
  note         = {Machine review of arXiv:1908.10747}
}
read the original abstract

"This paper introduces a new task and a new dataset", "we improve the state of the art in X by Y" -- it is rare to find a current natural language processing paper (or AI paper more generally) that does not contain such statements. What is mostly left implicit, however, is the assumption that this necessarily constitutes progress, and what it constitutes progress towards. Here, we make more precise the normally impressionistically used notions of language task and language game and ask how a research programme built on these might make progress towards the goal of modelling general language competence.

Figures

Figures reproduced from arXiv: 1908.10747 by the authors.

Figure 2
Figure 2. Three ways of making progress: improving models, improving datasets, improving tasks. (Not shown: [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 28 canonical work pages

  1. [1]

    Storrs Hall, Alexei Samsonovich, Matthias Scheutz, Matthew Schlesinger, Stuart C

    Sam Adams, Itmar Arel, Joscha Bach, Robert Coop, Rod Furlan, Ben Goertzel, J. Storrs Hall, Alexei Samsonovich, Matthias Scheutz, Matthew Schlesinger, Stuart C. Shapiro, and John Sowa. 2012. https://doi.org/10.1609/aimag.v33i1.2322 Mapping the Landscape of Human-Level Artificial General Intelligence . AI Magazine, 33(1):25--42

  2. [2]

    Stefanos Afantenos, Nicholas Asher, Farah Benamara, Anais Cadilhac, Cedric Degremont, Pascal Denis, Markus Guhe, Simon Keizer, Alex Lascarides, Oliver Lemon, Philippe Muller, Soumya Paul, Verena Rieser, and Laure Vieu. 2012. Developing a corpus of strategic conversation in the settlers of catan. In Proceedings of the 1st Workshop on Games and NLP, Kanazawa, Japan

  3. [3]

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko S \" u nderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018. http://arxiv.org/abs/1711.07280 Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments . In CVPR 2018

  4. [4]

    Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural Module Networks . In Proceedings of the 2016 Conference on Computer Vision and Pattern Recognition (CVPR 2016)

  5. [5]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In International Conference on Computer Vision (ICCV)

  6. [6]

    Ron Artstein and Massimo Poesio. 2008. https://doi.org/10.1136/bmj.312.7039.1166 Inter-Coder Agreement for Computational Linguistics . Computational Linguistic, 34(4):555--596

  7. [7]

    Marco Baroni, Armand Joulin, Allan Jabri, Germ \` a n Kruszewski, Angeliki Lazaridou, Klemen Simonic, and Tomas Mikolov. 2017 a . http://arxiv.org/abs/1701.08954 CommAI: Evaluating the first steps towards a useful general AI . arXiv, pages 1--9

  8. [8]

    Dowe, Katja Hofmann, Kristinn R

    Marco Baroni, Claes Stranneg rd, David L. Dowe, Katja Hofmann, Kristinn R. Th \' o risson, Jordi Bieger, Nader Chmait, Fernando Mart \' i nez-Plumed, and Jos \' e H \' e rnandez-Orallo. 2017 b . https://doi.org/10.1609/aimag.v38i3.2748 A New AI Evaluation Cosmos: Ready to Play the Game? AI Magazine, 38(3):66

Show all 45 references
  1. [9]

    Andrew Bennett and Max Shatkhin. 2018. Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction . In EMNLP 2018, pages 2667--2678

  2. [10]

    Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat, Hugo Larochelle, and Aaron Courville. 2017. http://arxiv.org/abs/1711.11017 HoME: a Household Multimodal Environment . ArXiv

  3. [11]

    Donna Byron, Alexander Koller, Jon Oberlander, Laura Stoia, and Kristina Striegnitz. 2007. Generating Instructions in Virtual Environments ( GIVE ): A Challenge and an Evaluation Testbed for NLG . In Proceedings of the Workshop on Shared Tasks and Comparative Evaluation in Nat...

  4. [12]

    Angel Chang, Angela Dai, Thomas Funkhouser, Manolis Savva, and Shuran Song. 2017. http://arxiv.org/abs/arXiv:1709.06158v1 Matterport3D : Learning from RGB-D Data in Indoor Environments . ArXiv

  5. [13]

    Cleo Condoravdi, Richard Crouch, Valeria de Paiva, Reinhard Stolle, and Daniel G Bobrow. 2003. https://doi.org/10.3115/1119239.1119245 Entailment, intensionality and text understanding . In Proc. of the HLT-NAACL 2003 Workshop on Text Meaning, pages 38--45

  6. [14]

    Marc-Alexandre C \^ o t \' e , \' A kos K \' a d \' a r, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. 2018. https://doi.org/EC.00123-10 [pii] 10.1128/EC.00123-10 TextWorld: A...

  7. [15]

    Danillo da Silva Rocha and Ivandr \' e Paraboni. 2016. Reference production in human-computer interaction : Issues for Corpus-based Referring Expression Generation . In LREC, pages 2994--2998

  8. [16]

    Hubert L. Dreyfus. 1981. From micro-worlds to knowledge: AI at an impasse. In John Haugeland, editor, Mind Design. MIT Press

  9. [17]

    Wallach, Hal Daum \' e III, and Kate Crawford

    Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daum \' e III, and Kate Crawford. 2018. http://arxiv.org/abs/1803.09010 Datasheets for datasets . CoRR, abs/1803.09010

  10. [18]

    Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. 2017. http://arxiv.org/abs/1712.03316 IQA: Visual Question Answering in Interactive Environments . ArXiv

  11. [19]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. http://arxiv.org/abs/1612.00837 Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering . In CVPR 2017

  12. [20]

    John Hewitt and Christopher D Manning. 2019. https://github.com/ A Structural Probe for Finding Syntax in Word Representations . In NAACL-HLT 2019

  13. [21]

    Nikolai Ilinykh, Sina Zarrie , and David Schlangen. 2018. The Task Matters. Comparing Image Captioning and Task-Based Dialogical Image Description . In Proceedings of 11th International Conference on Natural Language Generation (INLG 2018)

  14. [22]

    Allan Jabri, Armand Joulin, and Laurens van der Maaten. 2016. http://arxiv.org/abs/1606.08390 Revisiting Visual Question Answering Baselines . In European Conference on Computer Vision (ECCV)

  15. [23]

    Srini Janarthanam and Oliver Lemon. 2011. http://dl.acm.org/citation.cfm?id=2187717 The GRUVE Challenge : Generating Routes under Uncertainty in Virtual Environments . In ENLG '11 Proceedings of the 13th European Workshop on Natural Language Generation, pages 208--211

  16. [24]

    Lawrence Zitnick, and Ross Girshick

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. 2017. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning . In CVPR 2017, pages 1988----1997

  17. [25]

    Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. 2016. The malmo platform for artificial intelligence experimentation . IJCAI International Joint Conference on Artificial Intelligence, 2016-Janua:4246--4247

  18. [26]

    Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. 2019. http://arxiv.org/abs/1901.03035 Self-Monitoring Navigation Agent via Auxiliary Progress Estimation . ArXiv, pages 1--18

  19. [27]

    David Marr. 1982. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. W.H. Freeman, San Francisco, USA

  20. [28]

    Marvin Minsky and Seymour Papert. 1972. Progress Report on Artificial intelligence . Technical report, MIT Artificial Intelligence Laboratory, Cambridge, Mass., USA

  21. [29]

    Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. 2018. http://arxiv.org/abs/1809.00786 Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction . ArXiv

  22. [30]

    Karl Popper. 1934. Logik der Forschung. Mohr Siebeck

  23. [31]

    Sebastian Ruder. 2019. Neural Transfer Learning for Natural Language Processing . Ph.D. thesis, National University of Ireland, Galway

  24. [32]

    Chang, Alexey Dosovitskiy, Thomas Funkhouser, and Vladlen Koltun

    Manolis Savva, Angel X. Chang, Alexey Dosovitskiy, Thomas Funkhouser, and Vladlen Koltun. 2017. http://arxiv.org/abs/1712.03931 MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments . ArXiv, pages 1--14

  25. [33]

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. 2019. http://arxiv.org/abs/1904.01201 Habitat: A Platform for Embodied AI Research . ArXiv

  26. [34]

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George Van Den Driessche , Thore Graepel, and Demis Hassabis. 2017. h...

  27. [35]

    Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. 2017. http://yoavartzi.com/pub/slya-acl.2017.pdf A Corpus of Natural Language for Visual Reasoning . In Proceedings of the 2017 meeting of the Association for Computational Linguistics (ACL 2017)

  28. [36]

    Alane Suhr, Stephanie Zhou, Iris Zhang, Huajun Bai, and Yoav Artzi. 2018. A Corpus for Reasoning About Natural Language Grounded in Photographs . In Proceedings of NIPS 2018, Montreal, Canada

  29. [37]

    Bernard Suits. 1978. The Grasshopper: Games, Life, and Utopia. The University of Toronto Press, Toronto, Canada

  30. [38]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. 1998. Reinforcement Learning. MIT Press, Cambridge, USA

  31. [39]

    Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rockt \" a schel, Douwe Kiela, Arthur Szlam, and Jason Weston. 2019. http://arxiv.org/abs/1903.03094 Learning to Speak and Act in a Fantasy Text Adventure Game . ArXiv

  32. [40]

    Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. http://arxiv.org/abs/1704.05426 A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference . In Proceedings of the Conference of the North American Chapter of the Association for Computational Lingu...

  33. [41]

    Ludwig Wittgenstein. 1953/84. Tractatus Logicus Philosophicus und Philosophische Untersuchungen, volume 1 of Werkausgabe. Suhrkamp, Frankfurt am Main

  34. [42]

    Fei Xia, Amir Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. 2018. https://doi.org/arXiv:1808.10654v1 Gibson Env: Real-World Perception for Embodied Agents . In CVPR 2018

  35. [43]

    Takashi Yamauchi, Mikio Nakano, and Kotaro Funakoshi. 2013. A Robotic Agent in a Virtual Environment that Performs Situated Incremental Understanding of Navigational Utterances . In SIGdial 2013, August, pages 369--371

  36. [44]

    Claudia Yan, Dipendra Misra, Andrew Bennett, Aaron Walsman, Yonatan Bisk, and Yoav Artzi. 2018. http://arxiv.org/abs/arXiv:1801.07357v1 CHALET : Cornell House Agent Learning Environment . ArXiv

  37. [45]

    Dani Yogatama, Cyprien de Masson D'Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, and Phil Blunsom. 2019. http://arxiv.org/abs/1901.11373 Learning and Evaluating General Linguistic Intelligence . ArXiv,...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.