{"id":"e39c8119-5b37-485d-aefe-0aaea7cf3dad","arxiv_id":"2412.16970","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"RL environments should be described in user-friendly domain-specific languages or natural language, so non-engineers can define tasks and agents can generalize to new tasks.","lead":"This paper argues that the usual way of building reinforcement learning environments, with custom simulators written in code, blocks non-engineers from using RL and blocks agents from generalizing across tasks. It proposes a research agenda around user-friendly descriptions of environments, written in special languages or plain text, so that non-experts can define tasks and agents can read and reuse those descriptions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The agenda's usability and generalization payoffs both rest on Assumption 1 (DSLs/natural language are more user-friendly than programming), but the paper offers only Ludii anecdotes and no quantitative evidence; a novice user study is needed to see if non-engineers can write complete, compilable…","rationale":"The reader's weakest_assumption identifies the same linchpin: Assumption 1. My read agrees with the CONDITIONAL verdict. The paper is a position paper, so absence of proof is not a flaw by itself; the authors are transparent about the lack of quantitative evidence, and they provide a concrete Ludii example and cite external studies on AI adoption barriers (Section 3.4, [58,141]). However, the central claim—that the RL community should shift benchmark focus to user-friendly DSLs/natural languages—only becomes compelling if the user-friendliness premise holds for the intended audience. The self-cited Ludii library count and forum posts are weak support: game designers contributing to a Ludii forum are likely self-selected and may have technical affinity. The paper's own admission about GDL (Section 3.4) and the ambiguity concerns for natural language (Section 3.3) show the assumption is not trivially true. A single user study measuring whether non-programmers can produce complete, compilable descriptions would settle the matter more directly than any further theoretical argument. I therefore keep the verdict at UNCHANGED (CONDITIONAL) because the concern is exactly the one the reader flagged and does not warrant a harsher verdict for a research agenda.","tokens_in":25332,"tokens_out":4006,"duration_ms":36976,"concrete_test":"Run a user study with ~30 participants who have no programming experience. Ask each to describe three simple board games (e.g., Tic-Tac-Toe, a small blocking game, a racing game) in (a) Python with the Gym API, (b) Ludii's DSL, and (c) natural language. Measure: proportion of descriptions that compile/run; correctness against a reference implementation via automated unit tests; time on task; and self-reported difficulty. For the natural-language arm, also measure whether participants can successfully verify and correct an LLM-generated Ludii description (the workflow proposed in Section 3.3). If DSL/natural language descriptions are not correct at rates significantly higher than Python for novices, Assumption 1 is empirically undermined and the agenda's core premise fails. This is a single, feasible check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Assumption 1 (Section 3.4) is the linchpin: both the usability half (Section 3) and the generalization half (Section 4.3) require that people with little or no programming experience can author complete, unambiguous environment descriptions in a DSL or natural language. The paper's evidence is the Ludii design goal (Section 3.4), the >1400 game library, and third-party forum contributions; Section 6 explicitly concedes 'we have no quantitative evidence at this point' for the user-friendliness/generality trade-off. The admitted counterexample of logic-based GDL shows this is not automatically true. Moreover, natural language suffers from ambiguity and underspecification (Section 3.3), so the paper's proposed workflow of LLM translation plus human verification still requires users to read and fix DSL code, which demands the very expertise the agenda targets. If Assumption 1 fails for the intended end users, the Position loses its usability motivation and the zero-shot generalization thesis (which depends on complete, compilable descriptions as context) loses its foundational input.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This position paper argues that the customary practice of implementing RL environments in general-purpose programming languages imposes a usability barrier on non-engineer users and also obstructs progress on generalization, because there is no shared formalism in which different problems are represented. The authors advocate for a research agenda centered on describing environments in user-friendly domain-specific languages (DSLs) or natural language, such that (i) users with little programming expertise can formally describe their problems and (ii) algorithms can use the resulting complete descriptions as context to generalize zero-shot across all tasks describable in the chosen language. The paper states two explicit assumptions (Section 3.4), discusses potential issues such as action-space inference and simulation speed, notes other barriers to RL adoption (Section 5), and lays out desiderata and research directions (Section 6). It is an extended version of an earlier position paper, with additions covering other barriers and a more detailed agenda.","tokens_in":25498,"tokens_out":3359,"duration_ms":31297,"significance":"If the central position is correct, it would motivate a substantial shift in benchmark practice away from bespoke simulator code toward description-language-native environments, potentially democratizing RL for non-engineers and opening a new axis for zero-shot generalization research. The paper is honest and internally consistent: the two core assumptions are stated explicitly, the generalization claims in Section 4.3 are hedged, and Section 6 openly concedes the lack of quantitative evidence for the user-friendliness trade-off. It also credibly connects the agenda to concrete existing artifacts (Ludii, GAVEL, Ludax), gives a concrete desiderata list, and distinguishes its proposal from related DSL/context work. Its main weakness is that the load-bearing Assumption 1 rests on informal evidence and anecdotal experience rather than a user study; nevertheless the agenda is constructed so that this assumption could, in principle, be tested, which is a strength worth acknowledging.","major_comments":[{"comment":"Assumption 1 (that DSLs or natural language are more user-friendly than general-purpose programming languages for defining environments) is load-bearing for both halves of the agenda: the usability motivation in Section 3 and the generalization thesis in Section 4.3 both require non-engineers to be able to author complete, unambiguous descriptions. The only support offered is Ludii's design goal, the count of over 1400 game descriptions, and third-party forum contributions, while Section 6 admits \"we have no quantitative evidence at this point.\" Because the entire agenda collapses if this assumption fails for the intended end users, the manuscript should either include or cite a user study that measures whether non-programmers can successfully write and validate complete environment descriptions, or explicitly elevate this to the first, falsifiable item of the research agenda with pre-registered success criteria.","section":"Section 3.4 and Section 6"},{"comment":"The proposed natural-language workflow requires an LLM to translate the description into a DSL, after which \"a user can inspect the generated description and make corrections if necessary before it is compiled into a simulator.\" This verification-and-correction step still demands the ability to read and edit DSL code, which is precisely the expertise the agenda aims to remove. The paper should address how much DSL/verification competence is assumed of the end user and whether the verification step is realistically feasible for the target population; otherwise the usability advantage of natural-language descriptions is substantially weakened.","section":"Section 3.3"},{"comment":"The statement that complete environment descriptions \"are likely to be a prerequisite for unrestricted, zero-shot generalisation in RL\" is supported only by an analogy to humans learning new board games from rules, and the paper itself gives a video-game fire example where humans generalize without a complete description of the environment. This is not an internally inconsistent claim, but it is underspecified: the term \"unrestricted\" is never defined, and no concrete evidence or formal argument is given for why completeness is necessary rather than merely helpful. The claim should be reframed as a falsifiable hypothesis with a precise scope (e.g., across the set of tasks describable in a given DSL), and the authors should specify what experimental comparisons (such as context completeness versus zero-shot transfer on a DSL benchmark suite) would support or refute it.","section":"Section 4.3"}],"minor_comments":[{"comment":"The word \"exectuable\" should be \"executable.\"","section":"Section 3.4"},{"comment":"The phrase \"when action according to a policy\" should be \"when acting according to a policy.\"","section":"Section 2.1"},{"comment":"The Ludii example may confuse readers unfamiliar with the language because the comment says some rules are omitted as defaults without explaining what those defaults are; a short note on the default turn-taking and draw conditions would improve readability.","section":"Section 3.2"},{"comment":"The t-SNE figure (Fig. 2) is described only as \"reduced from a larger feature space\" with a citation; the caption could usefully state which features from [116] were used and how the embedding was computed.","section":"Section 6"},{"comment":"Reference [76] contains \"hum4n l4ngu4ge\" which appears to be either a deliberate obfuscation or a transcription error; if deliberate, the authors should add a note, as it may confuse readers.","section":"References"},{"comment":"The term \"unrestricted, zero-shot generalisation\" is used in Section 4.3 before being defined; a definition or at least a clarifying sentence in Section 4.1 would help.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The main concern is that the paper's evidence for its central assumption is heavily self-referential, coming mostly from the authors' own Ludii line of work and its follow-ups. This is not disqualifying for a position paper, but it makes the need for independent validation more acute. The revision should focus on making Assumption 1 testable and on softening or precisely delimiting the Section 4.3 prerequisite claim. I would not reject the paper on these grounds, as the agenda is coherent and the limitations are honestly stated, but the load-bearing nature of the assumption requires more than a footnote-level treatment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Key points you should know: this is a position paper, not a results paper. It argues that RL should move toward benchmarks where environments are described in user-friendly DSLs or natural language, with the description also serving as context for zero-shot generalisation. The central position is an extension of the authors' own prior paper, and they say so upfront. The new material is a structured agenda in Section 6 and a survey of other deployment barriers in Section 5. That's genuinely useful.\n\nWhat the paper does well: it states its two core assumptions explicitly in Section 3.4 and is unusually candid about the evidence gap. In Section 6 they write, 'we have no quantitative evidence at this point' for the user-friendliness/generality trade-off. That kind of honesty is rare and should be credited. The discussion of action-space inference difficulties in DSLs (Section 3.4) is concrete and informative, as is the recognition that natural language may need an LLM-translation step that still requires human verification.\n\nWhere the soft spots are: Assumption 1—that DSLs or natural language are more user-friendly than programming for the intended audience—is load-bearing for both halves of the agenda, but the support is anecdotal: Ludii's 1400+ games and third-party forum contributions. The stress-test concern is fair. Even a small pilot study with non-programmers writing descriptions would materially strengthen the case. Similarly, Section 4.3's claim that complete descriptions are a prerequisite for unrestricted zero-shot generalisation is argued by analogy to human board-game play; it is plausible but not demonstrated. These are not fatal flaws in a position paper, but they do mean the paper's impact depends on follow-up work.\n\nI disagree with the reader's downscore for self-citation: the paper extends the authors' own Ludii work, and self-citation is appropriate when the cited results are the direct prior foundation. The circularity burden is correctly low.\n\nWho this is for: researchers working on RL usability, benchmarking methodology, general game playing, and DSL design. It could frame a reading group on how RL evaluation should evolve. It deserves a serious referee: it is coherent, honest, and raises a question worth debating. I would send it to review as a position paper, with the caveat that the value is in the agenda, not a demonstrated result.","headline":"A clear, honest position paper that usefully spells out a research agenda for DSL/natural-language environment descriptions in RL—but its load-bearing assumption about user-friendliness remains quantitatively untested.","tokens_in":26091,"tokens_out":2617,"would_cite":true,"duration_ms":24337,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that reinforcement learning environments should be described in user-friendly domain-specific or natural languages, and that complete descriptions, supplied to agents as context, are the route to zero-shot generalization…","keywords":["reinforcement learning","environment description languages","domain-specific languages","natural language interfaces","zero-shot generalization","usability","benchmarks","contextual MDP"],"falsifier":"Run a controlled usability study in which people with no programming background describe the same dozen tasks (board games, simple control problems) in a user-friendly DSL, in natural language, and in a general-purpose language, then measure whether the DSL and natural-language descriptions are more accurate, complete, and faster to produce. If novices produce unusable or incomplete descriptions at comparable rates, the usability and generalisation arguments for the agenda collapse.","tokens_in":25084,"feed_emoji":"📝","tokens_out":4985,"duration_ms":42694,"temperature":0.7,"pith_summary":"Reinforcement learning today assumes that each new task is a bespoke simulator written by an engineer in a general-purpose programming language. This position paper argues that this assumption blocks two goals: letting people without programming expertise apply RL to their own problems, and letting agents generalize instantly to new problems. The proposed fix is to describe environments in user-friendly domain-specific languages or natural language, compile those descriptions into simulators, and hand the same descriptions to agents as context. If the agenda is right, benchmark practice shifts from hand-coded environments toward description-native ones, and non-engineers become first-class RL users. The paper states its central position as a call for more benchmarks with environments defined in user-friendly DSLs or natural languages.","feed_headline":"Describe RL tasks in plain language, not code","feed_subtitle":"The paper argues this would let non-engineers use reinforcement learning and help agents generalize to new tasks.","key_machinery":"The mechanism is the environment description as a dual-use object: a user-friendly formulation in a DSL or natural language that a compiler or language model translates into a runnable simulator, and that is simultaneously provided to the agent as context conditioning its policy or value function. Formalized as a contextual (PO)MDP or Markov game, this context must be complete enough to disambiguate between environments; incompleteness turns the collection of possible environments into a partially observable problem. The shared vocabulary of the description language is what makes the loop work in both directions: environments can be generated from contexts and contexts from environments, enabling procedural generation of training tasks and, in principle, zero-shot transfer to unseen descriptions.","core_discovery":"The paper's central claim is that the customary workflow—an engineer implementing each environment directly in a general-purpose programming language or a hardware-acceleration framework—is itself an obstacle to RL adoption and to generalization. It proposes that environments should be described in a shared, user-friendly language, with complete descriptions that can be compiled to executable simulators; those same descriptions, supplied to an agent as context, are argued to be a prerequisite for unrestricted zero-shot generalisation across every task expressible in the language. The paper supports this with a concrete example of a game description language that lets a user write tic-tac-toe as a short high-level script, and notes a library of over 1400 game descriptions contributed in part by non-programmers. It also identifies a practical rupture: succinct description languages often make it impossible to infer the full action space in advance, which violates a common assumption in deep RL APIs. The conclusion is a position statement: the RL community should place greater focus on benchmarks with environments defined in user-friendly DSLs or natural languages.","pith_inferences":["The agenda implicitly predicts a convergence between RL environment design and language-model-driven program synthesis: if natural-language descriptions become the interface, the reliability of translating language to simulators becomes a core RL benchmark question rather than a side concern.","A testable extension is to measure how much zero-shot transfer performance scales with the number and diversity of descriptions seen during pretraining; the paper does not make a scaling-law claim, but its argument suggests such a relationship.","The complete-description requirement may be too strong for physical-world tasks, where dynamics are not fully describable in language; the paper allows reward-only descriptions for such cases, which leaves a gap between virtual and physical generalization.","If the position is adopted, evaluation methodology must separate what an agent learned about the description language from what it learned about general RL competence, since performance on unseen descriptions could come from either.",""],"forward_implications":["Benchmark suites should be built around description languages rather than hand-written simulators, and evaluation should test agents on unseen descriptions in the same language.","Non-engineers—private individuals, small organisations, and domain experts—could specify their own tasks and receive a policy without writing code, provided the compiler and a sufficiently general agent exist.","Complete, compileable descriptions are claimed as a prerequisite for unrestricted zero-shot generalisation; partial contexts such as numeric goal coordinates or short instructions cannot do the same job.","Existing assumptions like knowing the full action space in advance may fail for user-friendly DSLs, so deep RL methods will need to handle action aliasing or variable action spaces.","Procedural generation of new descriptions in the same language can supply a curriculum, letting agents learn the semantics of the language and generalise across the whole describable space."],"supporting_citations":[{"why":"Supplies the standard Gym API that the paper identifies as entrenching programming-language environment definitions and the assumption of a known full action space.","marker":"[22]"},{"why":"Provides the user-friendly game description language and the library of over 1400 descriptions, the paper's main concrete evidence that non-programmers can write environments.","marker":"[115]"},{"why":"Is the earlier position paper that this work extends, establishing the usability-and-generalisation framing around environment descriptions.","marker":"[146]"},{"why":"Formalises contextual (PO)MDPs and zero-shot generalisation, which is the framework the paper uses for descriptions-as-context.","marker":"[66]"},{"why":"Shows existing benchmark contexts that are far from complete environment descriptions, serving as the contrast case for the paper's completeness claim.","marker":"[14]"},{"why":"Provides text-conditioned agents that generalise from instructions, the closest prior work to descriptions-as-context that still lacks compileable completeness.","marker":"[172]"},{"why":"Compiles a user-friendly board-game description language into hardware-accelerated simulation, addressing the tension between usability and simulation speed in the agenda.","marker":"[158]"},{"why":"Supplies the established claim that user-friendliness is a motivation for using domain-specific languages, underlying Assumption 1.","marker":"[95,8]"}],"fun_headline_variants":["Skip the code: describe RL tasks in plain language","RL for non-coders: describe tasks, not environments","Plain-language task descriptions could make RL usable and general","A shared DSL for RL tasks: no code required","Let users describe RL problems in words, not code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire agenda rests on the assumption that non-engineers can write complete, unambiguous environment descriptions in a DSL or natural language more easily than in code; the paper itself admits it has no quantitative evidence for this.","fun_headline_variants_meta":{"raw":{"variants":["Skip the code: describe RL tasks in plain language","RL for non-coders: describe tasks, not environments","Plain-language task descriptions could make RL usable and general","A shared DSL for RL tasks: no code required","Let users describe RL problems in words, not code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1555,"prompt_tokens":955,"completion_tokens":600,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":523}},"tokens_in":571,"tokens_out":600,"duration_ms":6110,"temperature":1.0,"reasoning_tokens":523,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:55:31.079129+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled usability study in which people with no programming background describe the same dozen tasks (board games, simple control problems) in a user-friendly DSL, in natural language, and in a general-purpose language, then measure whether the DSL and natural-language descriptions are more accurate, complete, and faster to produce. If novices produce unusable or incomplete descriptions at comparable rates, the usability and generalisation arguments for the agenda collapse.","supporting_citations":[{"cited_title":"In: Giacomo, G.D., Catala, A., Dilkina, B., Milano, M., Barro, S., Bugarín, A., Lang, J","cited_arxiv_id":null,"evidence_quote":"Provides the user-friendly game description language and the library of over 1400 descriptions, the paper's main concrete evidence that non-programmers can write environments."},{"cited_title":"In: Rocha, A.P., Steels, L., van den Herik, H.J","cited_arxiv_id":null,"evidence_quote":"Is the earlier position paper that this work extends, establishing the usability-and-generalisation framing around environment descriptions."},{"cited_title":"In: International Conference on Learning Repre- sentations (2020)","cited_arxiv_id":null,"evidence_quote":"Provides text-conditioned agents that generalise from instructions, the closest prior work to descriptions-as-context that still lacks compileable completeness."}],"review_version":1}