REVIEW 4 minor 45 references
Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research
T0 review · 0 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A new NLP task or dataset is only progress if it states which language capabilities it exercises; otherwise, the paper argues, the contribution is vacuous.
desk verdict A genuinely useful methodological critique that gives NLP researchers a cleaner vocabulary for task design, and the formal appendix does real work. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a set of formal tuple definitions — language task $(S, A, L, D_T)$, micro-world/environment $(S, A, E, R, D_W)$, interaction game $(P, A, o, T, E, D_G)$ with a disinterested player Nature — together with the 'involves' relation between a task and a capability subset. The definitions give a language to compare tasks, datasets, environments, and games, and the 'involves' relation supplies the yardstick: a contribution is progressive only when the claimed capability subset is specified precisely enough to be tested and is related to the overall capability set of a competent language user.
What would settle it
Run a large battery of diverse language tasks with human participants and factor-analyze the performance matrix. If no stable task-specific capability components emerge — that is, a single general language factor explains nearly all variance — then the paper's assumption that the overall capability set CL decomposes into task-linked subsets CT loses its empirical grounding.
Extended reading notes
Core claim
The central claim is that a new NLP task or dataset only constitutes progress if the relation between the task and the capabilities of a competent language user is stated explicitly; otherwise, the contribution is vacuous. The paper grounds this claim by defining a language task as a mapping from states to actions, at least one side involving natural language expressions, conforming to a task description; an environment as an action-to-state mapping conforming to a world description; and an interaction game as players, action spaces, an observability function, a turn-taking rule, an evaluation rule, and a game description. It then argues that any task motivation presupposes a set CL of capabilities of a competent language user and a subset CT that the task exercises. The strength of the motivation is ranked by how precisely CT is specified and by two dimensions: separability (whether CT can be handled independently of the rest of CL) and exhaustivity (whether the task exercises all of CT). A claim like 'the task involves the capability to compute syntactic structure' is strong because it could be wrong; a claim like 'the task involves the capability to do the task' is trivial.
Load-bearing premise
The argument depends on there being a well-defined set of capabilities of a competent language user that can be broken into subsets tied to individual tasks; if that decomposition does not exist, evaluating tasks by their claimed capability links has no foundation.
Editorial extensions
If this is right
- If the paper is right, a new task proposal should state, in falsifiable terms, which subset of competent-language-user capabilities the task engages, and how that subset relates to subsets of existing tasks.
- A dataset can be rejected as an unsatisfactory exemplification of a task when a model solves it without information the task description deems crucial, as happened when visual question answering was solved without visual input.
- Progress in tasks themselves means arguing that the new task's capability subset is more specific, more separable, or more exhaustive than the old one, not merely that a new dataset has been collected.
- Language games can be assessed by how close they come to unrestricted situated language interaction, the natural upper bound for language-game design.
- Probing results on trained models (for example, finding syntactic structure inside a network) should be read as evidence about whether a postulated capability is needed for the task, rather than as standalone architectural curiosities.
Reading between the lines
- A natural next step the author leaves implicit is a 'capability sheet' for each dataset or task, listing the claimed CT, its separability and exhaustivity, and the evidence that would falsify the claim; this would make the 'involves' relation contestable by inspection.
- If the capability-decomposition view is correct, then transfer learning provides a direct test: a model trained on a task that genuinely engages CT should improve performance on another task engaging the same CT; failures of transfer would indicate the capability claim was wrong.
- The framework could be applied to evaluate 'AI-complete' claims: saying a task is AI-complete amounts to claiming CT = CL, and by the paper's ranking that is the strongest and least likely kind of capability claim, not a casual rhetorical flourish.
- Operationalizing 'closeness to unrestricted situated language interaction' could yield a graded benchmark-construction guide (e.g., increasing the number of interacting players, hiding observations, or lengthening action sequences), although the paper does not specify such a metric.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This position paper argues that current NLP research habitually introduces new tasks, datasets, worlds, and games while leaving implicit what such contributions are progress toward. The author defines a language task as a mapping between input and output/action spaces involving natural language, a micro-world/environment as a mapping from actions to environmental responses, and an interaction game as a setting with repeated, connected language tasks regulated by turn-taking and evaluation rules. The central normative claim is that motivating a task or game requires making explicit claims about how the capabilities C_T it involves relate to the larger set C_L of capabilities of a competent language user; without such claims, a new task or dataset does not by itself constitute progress. The paper distinguishes three modes of progress (better models, better datasets, better tasks/games), analyzes their different argumentative burdens, and recommends a stronger connection to linguistics and cognitive psychology to make capability claims contestable. Formal definitions are provided in an appendix.
Significance. If the argument is accepted, it gives researchers a vocabulary for evaluating task and dataset contributions beyond benchmark scores, and it sharpens a critique that is often voiced only informally. The paper's main strength is that it makes an implicit argument structure explicit: progress claims rest on assumptions about the decomposition of language competence, and those assumptions should be stated so they can be challenged. The treatment of the VQA language-bias example and the discussion of dataset validity are concrete and instructive. The paper also honestly flags its own limitations, most notably in Section 3.1, where it acknowledges that motivating a task requires assumptions about how C_L decomposes. The paper contains no technical derivations or experiments, so its contribution is conceptual and methodological; it is a well-targeted piece of scientific criticism rather than a new empirical result. The central claim is conditional and remains coherent even without a fully specified theory of language capabilities.
minor comments (4)
- [Section 3.1] The ranking of capability claims from the trivial "task T involves the capability to do task T" to stronger falsifiable claims is intuitive, but the "involves" relation itself is never defined formally; a short worked example showing how a specific task would be paired with a non-trivial, falsifiable capability claim would make the central recommendation easier to apply.
- [Section 2.3 and Section 4.2] There are several typographical errors: "priviledged" in Section 2.3 should be "privileged", "Targetting" in Section 4.2 should be "Targeting", and "diagramm" in Section 4.3 should be "diagram".
- [Appendix A.1, Definition 1] The condition "with either the states in S or the actions in A (or both) having as part natural language expressions" is grammatically awkward and could be misread as requiring states or actions themselves to contain language expressions; a cleaner formulation would describe a function from states or actions to natural language expressions.
- [Section 4.3] The statement that tasks "are only grounded (to their left in the diagram) by capabilities" relies on the spatial layout of Figure 2; consider stating the direction of grounding textually without reference to the diagram's left side, since the figure may be rendered or read differently.
Circularity Check
No significant circularity identified: the paper is a methodological/normative framework with no fitted inputs or self-citation load-bearing derivation.
full rationale
No circularity found. The paper makes no empirical prediction and fits no parameters; its central claim is conditional and normative: contributions to NLP only count as progress toward modelling general language competence if their relation to a set C_L of capabilities is made explicit. The formal definitions in Appendix A are stipulative and do not presuppose the conclusion. The argument in Section 3.1 explicitly ranks capability claims from trivial ('task T involves the capability to do task T') to falsifiable strong claims, and concedes that motivating tasks requires assumptions about how C_L decomposes; this is an acknowledged limitation, not a circular step. The only self-citation (Ilinykh et al. 2018) is used as an external empirical pointer to production differences, not as the justification for the paper's framework. Hence the derivation chain is self-contained: no equation reduces to its input, and no prediction is renamed as a fit.
Assumptions & free parameters
assumptions (2)
- domain assumption There exists a set CL of capabilities of a competent language user.
- domain assumption For any task T, the set of required capabilities CT is a subset of CL, and CT can be characterized separately from the rest of CL.
Cite this review
Pith. "Pith review of Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research." pith.science (2026). https://pith.science/paper/CA6R4A4V
@misc{pith2026190810747,
author = {Pith},
title = {Pith review of: Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research},
year = {2026},
howpublished = {\url{https://pith.science/paper/CA6R4A4V}},
note = {Machine review of arXiv:1908.10747}
}
read the original abstract
"This paper introduces a new task and a new dataset", "we improve the state of the art in X by Y" -- it is rare to find a current natural language processing paper (or AI paper more generally) that does not contain such statements. What is mostly left implicit, however, is the assumption that this necessarily constitutes progress, and what it constitutes progress towards. Here, we make more precise the normally impressionistically used notions of language task and language game and ask how a research programme built on these might make progress towards the goal of modelling general language competence.
Figures
Reference graph
Works this paper leans on
-
[1]
Storrs Hall, Alexei Samsonovich, Matthias Scheutz, Matthew Schlesinger, Stuart C
Sam Adams, Itmar Arel, Joscha Bach, Robert Coop, Rod Furlan, Ben Goertzel, J. Storrs Hall, Alexei Samsonovich, Matthias Scheutz, Matthew Schlesinger, Stuart C. Shapiro, and John Sowa. 2012. https://doi.org/10.1609/aimag.v33i1.2322 Mapping the Landscape of Human-Level Artificial General Intelligence . AI Magazine, 33(1):25--42
-
[2]
Stefanos Afantenos, Nicholas Asher, Farah Benamara, Anais Cadilhac, Cedric Degremont, Pascal Denis, Markus Guhe, Simon Keizer, Alex Lascarides, Oliver Lemon, Philippe Muller, Soumya Paul, Verena Rieser, and Laure Vieu. 2012. Developing a corpus of strategic conversation in the settlers of catan. In Proceedings of the 1st Workshop on Games and NLP, Kanazawa, Japan
work page 2012
-
[3]
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko S \" u nderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018. http://arxiv.org/abs/1711.07280 Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments . In CVPR 2018
arXiv 2018
-
[4]
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural Module Networks . In Proceedings of the 2016 Conference on Computer Vision and Pattern Recognition (CVPR 2016)
work page 2016
-
[5]
Lawrence Zitnick, and Devi Parikh
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In International Conference on Computer Vision (ICCV)
2015
- [6]
-
[7]
Marco Baroni, Armand Joulin, Allan Jabri, Germ \` a n Kruszewski, Angeliki Lazaridou, Klemen Simonic, and Tomas Mikolov. 2017 a . http://arxiv.org/abs/1701.08954 CommAI: Evaluating the first steps towards a useful general AI . arXiv, pages 1--9
arXiv 2017
-
[8]
Dowe, Katja Hofmann, Kristinn R
Marco Baroni, Claes Stranneg rd, David L. Dowe, Katja Hofmann, Kristinn R. Th \' o risson, Jordi Bieger, Nader Chmait, Fernando Mart \' i nez-Plumed, and Jos \' e H \' e rnandez-Orallo. 2017 b . https://doi.org/10.1609/aimag.v38i3.2748 A New AI Evaluation Cosmos: Ready to Play the Game? AI Magazine, 38(3):66
Show all 45 references
-
[9]
Andrew Bennett and Max Shatkhin. 2018. Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction . In EMNLP 2018, pages 2667--2678
2018
-
[10]
Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat, Hugo Larochelle, and Aaron Courville. 2017. http://arxiv.org/abs/1711.11017 HoME: a Household Multimodal Environment . ArXiv
2017 arXiv
-
[11]
Donna Byron, Alexander Koller, Jon Oberlander, Laura Stoia, and Kristina Striegnitz. 2007. Generating Instructions in Virtual Environments ( GIVE ): A Challenge and an Evaluation Testbed for NLG . In Proceedings of the Workshop on Shared Tasks and Comparative Evaluation in Nat...
2007
-
[12]
Angel Chang, Angela Dai, Thomas Funkhouser, Manolis Savva, and Shuran Song. 2017. http://arxiv.org/abs/arXiv:1709.06158v1 Matterport3D : Learning from RGB-D Data in Indoor Environments . ArXiv
2017 arXiv
-
[13]
Cleo Condoravdi, Richard Crouch, Valeria de Paiva, Reinhard Stolle, and Daniel G Bobrow. 2003. https://doi.org/10.3115/1119239.1119245 Entailment, intensionality and text understanding . In Proc. of the HLT-NAACL 2003 Workshop on Text Meaning, pages 38--45
2003
-
[14]
Marc-Alexandre C \^ o t \' e , \' A kos K \' a d \' a r, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. 2018. https://doi.org/EC.00123-10 [pii] 10.1128/EC.00123-10 TextWorld: A...
2018 doi
-
[15]
Danillo da Silva Rocha and Ivandr \' e Paraboni. 2016. Reference production in human-computer interaction : Issues for Corpus-based Referring Expression Generation . In LREC, pages 2994--2998
2016
-
[16]
Hubert L. Dreyfus. 1981. From micro-worlds to knowledge: AI at an impasse. In John Haugeland, editor, Mind Design. MIT Press
1981
-
[17]
Wallach, Hal Daum \' e III, and Kate Crawford
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daum \' e III, and Kate Crawford. 2018. http://arxiv.org/abs/1803.09010 Datasheets for datasets . CoRR, abs/1803.09010
2018 arXiv
-
[18]
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. 2017. http://arxiv.org/abs/1712.03316 IQA: Visual Question Answering in Interactive Environments . ArXiv
2017 arXiv
-
[19]
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. http://arxiv.org/abs/1612.00837 Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering . In CVPR 2017
2017 arXiv
-
[20]
John Hewitt and Christopher D Manning. 2019. https://github.com/ A Structural Probe for Finding Syntax in Word Representations . In NAACL-HLT 2019
2019
-
[21]
Nikolai Ilinykh, Sina Zarrie , and David Schlangen. 2018. The Task Matters. Comparing Image Captioning and Task-Based Dialogical Image Description . In Proceedings of 11th International Conference on Natural Language Generation (INLG 2018)
2018
-
[22]
Allan Jabri, Armand Joulin, and Laurens van der Maaten. 2016. http://arxiv.org/abs/1606.08390 Revisiting Visual Question Answering Baselines . In European Conference on Computer Vision (ECCV)
2016 arXiv
-
[23]
Srini Janarthanam and Oliver Lemon. 2011. http://dl.acm.org/citation.cfm?id=2187717 The GRUVE Challenge : Generating Routes under Uncertainty in Virtual Environments . In ENLG '11 Proceedings of the 13th European Workshop on Natural Language Generation, pages 208--211
2011
-
[24]
Lawrence Zitnick, and Ross Girshick
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. 2017. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning . In CVPR 2017, pages 1988----1997
2017
-
[25]
Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. 2016. The malmo platform for artificial intelligence experimentation . IJCAI International Joint Conference on Artificial Intelligence, 2016-Janua:4246--4247
2016
-
[26]
Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. 2019. http://arxiv.org/abs/1901.03035 Self-Monitoring Navigation Agent via Auxiliary Progress Estimation . ArXiv, pages 1--18
2019 arXiv
-
[27]
David Marr. 1982. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. W.H. Freeman, San Francisco, USA
1982
-
[28]
Marvin Minsky and Seymour Papert. 1972. Progress Report on Artificial intelligence . Technical report, MIT Artificial Intelligence Laboratory, Cambridge, Mass., USA
1972
-
[29]
Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. 2018. http://arxiv.org/abs/1809.00786 Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction . ArXiv
2018 arXiv
-
[30]
Karl Popper. 1934. Logik der Forschung. Mohr Siebeck
1934
-
[31]
Sebastian Ruder. 2019. Neural Transfer Learning for Natural Language Processing . Ph.D. thesis, National University of Ireland, Galway
2019
-
[32]
Chang, Alexey Dosovitskiy, Thomas Funkhouser, and Vladlen Koltun
Manolis Savva, Angel X. Chang, Alexey Dosovitskiy, Thomas Funkhouser, and Vladlen Koltun. 2017. http://arxiv.org/abs/1712.03931 MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments . ArXiv, pages 1--14
2017 arXiv
-
[33]
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. 2019. http://arxiv.org/abs/1904.01201 Habitat: A Platform for Embodied AI Research . ArXiv
2019 arXiv
-
[34]
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George Van Den Driessche , Thore Graepel, and Demis Hassabis. 2017. h...
2017 doi
-
[35]
Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. 2017. http://yoavartzi.com/pub/slya-acl.2017.pdf A Corpus of Natural Language for Visual Reasoning . In Proceedings of the 2017 meeting of the Association for Computational Linguistics (ACL 2017)
2017
-
[36]
Alane Suhr, Stephanie Zhou, Iris Zhang, Huajun Bai, and Yoav Artzi. 2018. A Corpus for Reasoning About Natural Language Grounded in Photographs . In Proceedings of NIPS 2018, Montreal, Canada
2018
-
[37]
Bernard Suits. 1978. The Grasshopper: Games, Life, and Utopia. The University of Toronto Press, Toronto, Canada
1978
-
[38]
Sutton and Andrew G
Richard S. Sutton and Andrew G. Barto. 1998. Reinforcement Learning. MIT Press, Cambridge, USA
1998
-
[39]
Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rockt \" a schel, Douwe Kiela, Arthur Szlam, and Jason Weston. 2019. http://arxiv.org/abs/1903.03094 Learning to Speak and Act in a Fantasy Text Adventure Game . ArXiv
2019 arXiv
-
[40]
Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. http://arxiv.org/abs/1704.05426 A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference . In Proceedings of the Conference of the North American Chapter of the Association for Computational Lingu...
2018 arXiv
-
[41]
Ludwig Wittgenstein. 1953/84. Tractatus Logicus Philosophicus und Philosophische Untersuchungen, volume 1 of Werkausgabe. Suhrkamp, Frankfurt am Main
1953
-
[42]
Fei Xia, Amir Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. 2018. https://doi.org/arXiv:1808.10654v1 Gibson Env: Real-World Perception for Embodied Agents . In CVPR 2018
2018 arXiv
-
[43]
Takashi Yamauchi, Mikio Nakano, and Kotaro Funakoshi. 2013. A Robotic Agent in a Virtual Environment that Performs Situated Incremental Understanding of Navigational Utterances . In SIGdial 2013, August, pages 369--371
2013
-
[44]
Claudia Yan, Dipendra Misra, Andrew Bennett, Aaron Walsman, Yonatan Bisk, and Yoav Artzi. 2018. http://arxiv.org/abs/arXiv:1801.07357v1 CHALET : Cornell House Agent Learning Environment . ArXiv
2018 arXiv
-
[45]
Dani Yogatama, Cyprien de Masson D'Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, and Phil Blunsom. 2019. http://arxiv.org/abs/1901.11373 Learning and Evaluating General Linguistic Intelligence . ArXiv,...
2019 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.