Pith. sign in

REVIEW 2 major objections 5 minor 23 references

Under a deterministic launch check, self-distillation compounds clean game generation on unseen families from 8.8% to 42.2% and full best-of-K coverage—because the verifier, not the optimizer, sets the curriculum.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 17:24 UTC pith:LNCAOZ34

load-bearing objection Clean causal isolation of verifier precision in self-distillation: launch-gated RFT compounds cross-family Godot generation, gold-dup regresses, BUILD-swap erases the gain. the 2 major comments →

arxiv 2607.09709 v1 pith:LNCAOZ34 submitted 2026-06-23 cs.AI cs.SE

The Verifier is the Curriculum: Execution-Gated Self-Distillation for Cross-Family Game Generation

Chenyu Zhou , Qiliang Jiang , Shuning Wu , Xu Zhou This is my paper
classification cs.AI cs.SE
keywords self-distillationrejection samplinggame generationverifier precisionreward hackingexecution verificationGodotcross-family generalization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Post-training a code generator against a learned judge can reward cheap surface tricks that raise the score without making the artifact better. This paper trains against the opposite signal: a fixed, judge-free engine check that only accepts a generated Godot project if it launches cleanly headless, with no parse, load, or runtime error. On a from-scratch game-generation benchmark, three rounds of rejection-sampling self-distillation under that gate lift clean generation on four held-out game families from 8.8% to 42.2% per candidate and push best-of-K coverage to the gold ceiling of 25/25. Matched controls show the gain is not from adding more gold data or from the training loop itself: exact gold duplication hurts, and swapping only the filter for a near-vacuous build check erases the improvement. A second execution-grounding probe rises in step, arguing the accepted projects are functional rather than empty stubs. The lesson the authors press is simple and general: in self-distillation, what the verifier certifies is what the model learns.

Core claim

When rejection-sampling self-distillation is gated by a deterministic, ungameable strict-launch check, out-of-family clean game generation compounds across rounds; the filter’s precision is causal, because an exactly matched gold-duplication control regresses below the base model and a lenient BUILD filter swap returns performance to base, isolating the signal rather than the optimizer or data volume.

What carries the argument

Strict-launch: a binary, judge-free headless engine predicate (exit code 0 and no parse/load/runtime error) used as the acceptance filter in iterative self-distillation, so only projects that actually run become the next training distribution—the mechanism behind the claim that “the verifier is the curriculum.”

Load-bearing premise

That a clean headless launch, backed by a secondary execution-grounding score, is precise enough to certify functional transferable projects rather than harness-specific or launch-but-empty artifacts.

What would settle it

Rerun the same loop on the same split with only the acceptance filter changed to another high-throughput but low-precision check (or measure accepted projects with a stronger independent functional audit); if compounding still appears under a high false-positive gate, or if launch-clean projects systematically fail richer playability checks, the precision-as-curriculum claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Self-improvement loops for open-ended generation are limited by verifier integrity, not by data volume alone.
  • Exact duplication of gold references can regress capability relative to diverse self-generated clean fuel of the same size.
  • A binary ungameable execution gate can act as a coverage process that widens the set of working solutions across rounds.
  • Game generation is a measurable testbed for studying how filter precision shapes transfer under a fixed optimizer.
  • Replacing a gameable learned judge with a deterministic launch gate changes what post-training transfers from cosmetic score features to executable behavior.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same precision-over-volume principle should matter for other conjunctive synthesis tasks—full web apps, infrastructure configs, or multi-file agents—where partial correctness is not enough to run.
  • If the ceiling of self-improvement is set by the verifier, investing in harder ungameable execution checks may raise capability more than scaling soft preference models alone.
  • Learned judges may remain useful as secondary scorers only when an ungameable execution gate already filters out non-running candidates from below.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper argues that post-training a code generator against a deterministic, judge-free execution filter (strict-launch: clean headless Godot launch with exit code 0 and no parse/load/runtime error) yields compounding cross-family gains under rejection-sampling self-distillation, whereas a learned visual judge is gameable. On GameCraft-Bench, a Qwen3-14B+LoRA model trained only on ten families improves clean generation on four held-out families from 8.8% to 42.2% per-candidate and best-of-K coverage from 18/25 to 25/25 over three rounds. Matched controls attribute the gain to verifier precision and fuel diversity/quality rather than data volume or the optimizer: exact gold-duplication regresses below the base model; a count-matched generator swap isolates comparable quality and quantity channels; and swapping only the filter for the near-vacuous BUILD check erases the first-round gain. A secondary execution-grounding probe and a static code audit are used to argue that accepted projects are functional rather than launch-but-empty stubs. The stated lesson is that in self-distillation the verifier is the curriculum.

Significance. If the result holds, the paper supplies unusually clean causal evidence that the precision and ungameability of an acceptance filter, not merely the self-distillation optimizer or added sample count, determine what capability transfers. The design is a genuine strength: budget- and task-matched controls, a filter-swap ablation, paired task-level cluster-permutation tests (N=25), leave-one-family-out checks on the key contrast, training-seed robustness for round 1, and an independent calibrated grounding signal. Framing game generation as a verifiable testbed for a falsifiable principle (precision and exploration conditions) is useful beyond the specific engine. The contribution is empirical and methodological rather than a new algorithm; its value is the controlled isolation of the verifier as the causal variable.

major comments (2)
  1. [§8, §10, abstract] The central empirical claim about compounding clean-launch rates and the causal role of filter precision is well supported by Table 1 and the matched controls in §7. The interpretive claim that the transferred capability is functional game generation, however, rests on a thinner bridge. §8’s static audit (more GDScript lines, signals, files) and exec-ground (SceneTree state change under passive/synthetic/replay probes, score ≥0.50) rule out empty stubs and pure non-execution, but neither checks brief- or family-specific mechanics, win conditions, or input semantics beyond the probe. The paper itself calls exec-ground “a functional lower bound, not a measure of playability” (§10). As written, the curriculum may primarily teach engine-compliant project structure that launches and exhibits probeable state change. This does not invalidate the launch-rate results or the BUILD-swap isolation o
  2. [§6, §10, §11] The general principle that “the ceiling of a self-improvement loop is set by the integrity of its verifier” (§11) is motivated by a single 14B LoRA setup, one engine, and four held-out families (N=25 tasks). §10 acknowledges broader families and larger models as next tests, and training-seed robustness is reported only for round 1 (§6). That is acceptable for a testbed paper, but the conclusion currently states the principle more strongly than the evidence warrants. Please qualify the conclusion so that the load-bearing claim remains the controlled filter-swap result on this benchmark, with the broader implication framed as a hypothesis supported by one instance rather than established.
minor comments (5)
  1. [abstract, §6] Notation for p-values is inconsistent in the abstract and body (e.g., p<1e-4 vs. p <10 −4 with a space and superscript). Standardize to a single scientific notation throughout.
  2. [Figure 1, §4] Figure 1’s verification ladder is helpful; a one-line quantitative callout of the BUILD vs. strict-launch acceptance rates (99.9% vs. 8.2% on the supervised training-task pool, as stated in §4) on the figure itself would make the precision gap immediately readable.
  3. [Algorithm 1, §5] Algorithm 1 mentions “at least three files” as an acceptance side-condition; the main text should briefly justify this threshold (or note it as a stub filter) so readers do not treat it as an unmotivated hyperparameter.
  4. [§3, Appendix A] The companion diagnosis that the visual judge is gameable is important for motivation (§3, Appendix A) but is summarized from separate work. A short table of the asset-swap score deltas in the main text would make the “why not train against the judge” argument self-contained without forcing a full appendix read.
  5. [§9] Related work (§9) correctly positions the paper relative to STaR/ReST and reward-hacking literature. A sentence distinguishing this filter-swap design from concurrent runtime-verification game-generation work (already cited) would further clarify the novelty axis: not a new game generator, but a controlled study of verifier integrity under fixed optimizer.

Circularity Check

0 steps flagged

No significant circularity: empirical self-distillation results on held-out families with independent gold ceiling and filter-swap controls; no prediction reduces to a fitted input by construction.

full rationale

The paper’s load-bearing chain is experimental, not definitional. Round-k models are trained on gold plus self-generated projects that pass a fixed engine predicate (strict-launch); evaluation is clean-launch rate and best-of-K coverage on four families never used in training or fuel harvest, with an independent gold ceiling of 25/25. That the training filter and primary metric share the same engine check is standard rejection-sampling practice, not circularity: the non-trivial claims (compounding transfer, quality vs quantity channels, causal role of filter precision) are tested by matched controls that can and do fail—exact gold-duplication regresses below SFT, and swapping only the filter for near-vacuous BUILD erases the gain. Exec-ground is a secondary, non-training signal calibrated on gold vs known-bad projects. The companion diagnosis that the visual judge is gameable is motivational context in §3/Appendix A and is not required for the compounding or filter-swap results. There is no fitted parameter renamed as a prediction, no uniqueness theorem imported from the authors, and no equation that reduces a claimed derivation to its inputs. Concerns about whether launch+grounding certify brief-specific playability are validity/scope issues, not circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

The central claim is empirical and rests on operational definitions of the filters, a family-based train/held-out split, standard LoRA fine-tuning choices, and the assumption that headless Godot behavior is a stable, ungameable predicate. No new physical entities are postulated; free parameters are training and scoring hyperparameters; axioms are domain assumptions about the engine and the benchmark split.

free parameters (4)
  • LoRA rank r and alpha = r=16, α=32
    Chosen as r=16, α=32; standard but not derived; affects capacity of each round’s adapter.
  • learning rate and epochs = 1e-4, 8 epochs
    lr=1e-4, 8 epochs per round; hand-chosen training schedule that defines each Mt+1.
  • samples per brief K and keep-cap = K=8, keep≤2, ≥3 files
    K=8 candidates per brief; keep up to 2 deduplicated clean candidates with ≥3 files; directly shapes fuel size (67/123/186).
  • exec-ground score weights and threshold = 0.35/0.25/0.40, threshold 0.50
    score = 1[error-free]·(0.35 s_state + 0.25 s_passive + 0.40 s_active), grounded if ≥0.50; weights and threshold chosen and calibrated on 15 gold / 16 negatives.
axioms (5)
  • domain assumption Headless Godot strict-launch (exit 0, no parse/load/runtime error on stderr) is a deterministic, judge-free, ungameable predicate of the artifact under a fixed engine.
    Stated in §3–§4 and Appendix B; load-bearing for the claim that the filter cannot be gamed by surface features.
  • domain assumption The lenient BUILD check (open headless and quit) is near-vacuous and accepts nearly all candidates including parse-broken projects (99.9% / 887 of 888).
    Empirical measurement in §4 used to justify the filter-swap control in §7.
  • domain assumption Family-based partition (train on 10 families / 111 gold; hold out horror, rhythm, puzzle, shooter / 25 tasks) measures cross-family generalization with zero held-out leakage into fuel.
    §2 setup; authors state verification of zero leakage; underpins the out-of-family claim.
  • domain assumption Standard rejection-sampling self-distillation with LoRA fine-tuning from the base each round is a fixed optimizer, so differences across controls are attributable to the acceptance filter or fuel content.
    §5 method; enables causal attribution in §7.
  • standard math Cluster-permutation tests with task as resampling unit (N=25) and Wilson intervals are appropriate for the paired per-candidate and coverage comparisons.
    §5 evaluation protocol and Appendix C.
invented entities (3)
  • strict-launch gate independent evidence
    purpose: Binary acceptance filter for self-distillation fuel: clean headless launch with no parse/load/runtime error.
    Operational definition introduced as the training gate; calibrated against gold (25/25 pass) and BUILD-passing failures.
  • execution-grounding (exec-ground) signal independent evidence
    purpose: Independent functional lower bound that instruments SceneTree state change under passive/synthetic/replay modes.
    Secondary ungameable metric used only for audit (§8), not training; calibrated on gold vs known-bad projects.
  • verification ladder no independent evidence
    purpose: Ordered ranking of signals (BUILD < strict-launch < exec-ground < multimodal judge) by certification vs cost/gameability.
    Conceptual framing in §4/Figure 1 that organizes the experimental design; not an ontological claim.

pith-pipeline@v1.1.0-grok45 · 17850 in / 3745 out tokens · 41348 ms · 2026-07-14T17:24:05.496786+00:00 · methodology

0 comments
read the original abstract

Post-training a code generator against a learned judge can optimize proxy features that raise the score without improving the artifact. We study the opposite signal: a deterministic, judge-free, ungameable filter -- whether a generated project launches cleanly under a headless engine (strict-launch). Under this gate, rejection-sampling self-distillation compounds out-of-family generalization. On GameCraft-Bench (mapping a natural-language brief to a complete Godot project), a 14B model (Qwen3-14B+LoRA) distilled under strict-launch raises clean generation on four unseen game families from 8.8% to 42.2% per-candidate and best-of-K coverage from 18/25 to 25/25 (the gold ceiling) over three rounds, each a significant gain (p=0.0019, p<1e-4, p<1e-4). The gain is not from merely adding data: an exactly-matched gold-duplication control regresses below the base model (5.6% vs. 8.8%, p=0.019), while a count-matched decomposition splits the round-1-to-2 jump into comparable quality (+8.8pp) and quantity (+8.5pp) channels. Most directly, rerunning the loop with only the filter swapped -- the lenient BUILD check, which passes 99.9% of generations, in place of the launch gate -- erases the gain entirely (back to base, p=1e-3 vs. the launch-gated round), isolating verifier precision rather than the optimizer. A second ungameable signal, headless execution grounding, rises monotonically across rounds and yields far more grounded candidates than gold-duplication at a matched budget (16 vs. 5), confirming the gains are functional, not launch-but-empty. Game generation is a verifiable testbed for one lesson: the verifier is the curriculum -- what it certifies is what the model learns.

Figures

Figures reproduced from arXiv: 2607.09709 by Chenyu Zhou, Qiliang Jiang, Shuning Wu, Xu Zhou.

Figure 1
Figure 1. Figure 1: The verification ladder, ordered by how much each signal certifies. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Three rounds compound. per-candidate clean-launch on unseen families (blue) climbs [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Same +67-sample budget; only the high-precision filter compounds. Neither control that [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Execution grounding rises monotonically: held-out tasks with at least one grounded [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 20 linked inside Pith

  1. [1]

    Concrete problems in AI safety.arXiv preprint arXiv:1606.06565, 2016

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man´ e. Concrete problems in AI safety.arXiv preprint arXiv:1606.06565, 2016. URLhttps://arxiv. org/abs/1606.06565

    Pith/arXiv arXiv 2016

  2. [2]

    GameDevBench: Evaluating agentic capabilities through game develop- ment

    Wayne Chi, Yixiong Fang, Arnav Yayavaram, Siddharth Yayavaram, Seth Karten, Qi- uhong Anna Wei, Runkun Chen, Alexander Wang, Valerie Chen, Ameet Talwalkar, and Chris Donahue. GameDevBench: Evaluating agentic capabilities through game develop- ment. InInternational Conference on Machine Learning (ICML), 2026. URLhttps: //arxiv.org/abs/2602.11103

    Pith/arXiv arXiv 2026

  3. [3]

    Benchmarking reward hack de- tection in code environments via contrastive analysis.arXiv preprint arXiv:2601.20103, 2026

    Darshan Deshpande, Anand Kannappan, and Rebecca Qian. Benchmarking reward hack de- tection in code environments via contrastive analysis.arXiv preprint arXiv:2601.20103, 2026. URLhttps://arxiv.org/abs/2601.20103. Introduces the TRACE benchmark

    arXiv 2026

  4. [4]

    Reinforced self-training (ReST) for lan- guage modeling.arXiv preprint arXiv:2308.08998, 2023

    Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Ab- hishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. Reinforced self-training (ReST) for lan- guage modeling.arXiv preprint arXiv:2308.08998, 2023. URLhttps://arxiv.org/abs/ 2308.08998. 11

    Pith/arXiv arXiv 2023

  5. [5]

    V-STaR: Training verifiers for self-taught reasoners.arXiv preprint arXiv:2402.06457, 2024

    Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-STaR: Training verifiers for self-taught reasoners.arXiv preprint arXiv:2402.06457, 2024. URLhttps://arxiv.org/abs/2402.06457

    Pith/arXiv arXiv 2024

  6. [6]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. URLhttps://arxiv. org/abs/2106.09685

    Pith/arXiv arXiv 2022

  7. [7]

    GameGen-Verifier: Parallel keypoint-based verification for LLM- generated games via runtime state injection.arXiv preprint arXiv:2605.07442, 2026

    Chaobo Jia, Ruipeng Wan, Ting Sun, Weihao Tan, Borui Wan, Yuxuan Tong, Guangming Sheng, and Hong Xu. GameGen-Verifier: Parallel keypoint-based verification for LLM- generated games via runtime state injection.arXiv preprint arXiv:2605.07442, 2026. URL https://arxiv.org/abs/2605.07442

    Pith/arXiv arXiv 2026

  8. [8]

    OpenGame: Open agentic coding for games.arXiv preprint arXiv:2604.18394, 2026

    Yilei Jiang, Jinyuan Hu, Qianyin Xiao, Yaozhi Zheng, Ruize Ma, Kaituo Feng, Jiaming Han, Tianshuo Peng, Kaixuan Fan, Manyuan Zhang, and Xiangyu Yue. OpenGame: Open agentic coding for games.arXiv preprint arXiv:2604.18394, 2026. URLhttps://arxiv.org/abs/ 2604.18394. Introduces the GameCoder-27B code model and the OpenGame-Bench bench- mark

    Pith/arXiv arXiv 2026

  9. [9]

    Ziegler, Elizabeth Barnes, and Lawrence Chan

    Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes, and Lawrence ...

    Pith/arXiv arXiv 2025

  10. [10]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles (SOSP), 2023. URLhttps://arxiv.org/abs/2309.06180

    Pith/arXiv arXiv 2023

  11. [11]

    GameCraft-Bench: Can agents build playable games end-to-end in a real game engine?arXiv preprint arXiv:2606.17861, 2026

    Tongxu Luo, Rongsheng Wang, Jiaxi Bi, Chenming Xu, Zhengyang Tang, Jianlong Chen, Juhao Liang, Ke Ji, Shuqi Guo, Yuhao Du, Fan Bu, Wenyu Du, Xiaotong Zhang, Kyle Li, Shaobo Wang, Linfeng Zhang, Yuxuan Liu, Xin Lai, Chenxin Li, Yiduo Guo, Zhexin Zhang, Xinyuan Wang, Tianyi Bai, Ziniu Li, and Benyou Wang. GameCraft-Bench: Can agents build playable games end...

    Pith/arXiv arXiv 2026

  12. [12]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Qwen Team. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. URLhttps: //arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  13. [13]

    Co-Reyes, Rishabh Agarwal, et al

    Avi Singh, John D. Co-Reyes, Rishabh Agarwal, et al. Beyond human data: Scaling self- training for problem-solving with language models.Transactions on Machine Learning Re- search (TMLR), 2024. URLhttps://arxiv.org/abs/2312.06585. Introduces ReST EM ; first appeared as arXiv:2312.06585

    Pith/arXiv arXiv 2024

  14. [14]

    Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. InAdvances in Neural Information Processing Systems (NeurIPS), 2022. URLhttps://arxiv.org/abs/2209.13085. 12

    Pith/arXiv arXiv 2022

  15. [15]

    JAMER: Project-level code framework dataset and benchmark on professional game engines.arXiv preprint arXiv:2606.19830, 2026

    Jianwen Sun, Chuanhao Li, Zizhen Li, Yukang Feng, Fanrui Zhang, Yifei Huang, Yu Dai, and Kaipeng Zhang. JAMER: Project-level code framework dataset and benchmark on professional game engines.arXiv preprint arXiv:2606.19830, 2026. URLhttps://arxiv.org/abs/2606. 19830

    Pith/arXiv arXiv 2026

  16. [16]

    Reward hacking benchmark: Measuring exploits in LLM agents with tool use.arXiv preprint arXiv:2605.02964, 2026

    Kunvar Thaman. Reward hacking benchmark: Measuring exploits in LLM agents with tool use.arXiv preprint arXiv:2605.02964, 2026. URLhttps://arxiv.org/abs/2605.02964

    Pith/arXiv arXiv 2026

  17. [17]

    Scaling relationship on learning mathematical reasoning with large language models.arXiv preprint arXiv:2308.01825, 2023

    Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models.arXiv preprint arXiv:2308.01825, 2023. URLhttps://arxiv.org/abs/ 2308.01825

    Pith/arXiv arXiv 2023

  18. [18]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. STaR: Bootstrapping reasoning with reasoning. InAdvances in Neural Information Processing Systems (NeurIPS), volume 35, pages 15476–15488, 2022. URLhttps://arxiv.org/abs/2203.14465

    Pith/arXiv arXiv 2022

  19. [19]

    Embarrassingly simple self-distillation improves code generation.arXiv preprint arXiv:2604.01193, 2026

    Ruixiang Zhang, Richard He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, and Yizhe Zhang. Embarrassingly simple self-distillation improves code generation.arXiv preprint arXiv:2604.01193, 2026. URLhttps://arxiv.org/abs/2604.01193

    Pith/arXiv arXiv 2026

  20. [20]

    V-GameGym: Visual game generation for code large language models.arXiv preprint arXiv:2509.20136, 2025

    Wei Zhang, Jack Yang, Renshuai Tao, Lingzheng Chai, Shawn Guo, Jiajun Wu, Xiaoming Chen, Ganqu Cui, Ning Ding, Xander Xu, Hu Wei, and Bowen Zhou. V-GameGym: Visual game generation for code large language models.arXiv preprint arXiv:2509.20136, 2025. URL https://arxiv.org/abs/2509.20136

    arXiv 2025

  21. [21]

    Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734, 2026

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734, 2026. URLhttps://arxiv.org/abs/2601.18734

    Pith/arXiv arXiv 2026

  22. [22]

    Cheating auto- matic LLM benchmarks: Null models achieve high win rates

    Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Jing Jiang, and Min Lin. Cheating auto- matic LLM benchmarks: Null models achieve high win rates. InInternational Conference on Learning Representations (ICLR), 2025. URLhttps://arxiv.org/abs/2410.07137. Oral

    Pith/arXiv arXiv 2025

  23. [23]

    ImpossibleBench: Measuring LLMs’ propensity of exploiting test cases.arXiv preprint arXiv:2510.20270, 2025

    Ziqian Zhong, Aditi Raghunathan, Nicholas Carlini, et al. ImpossibleBench: Measuring LLMs’ propensity of exploiting test cases.arXiv preprint arXiv:2510.20270, 2025. URLhttps: //arxiv.org/abs/2510.20270. A The companion diagnosis study Section 3 summarizes the evidence that the benchmark’s learned visual judge is gameable. This appendix gives the details,...

    arXiv 2025