Pith. sign in

REVIEW 3 major objections 4 minor 34 references

CRAFT: Learn the Schema, Execute the Plan

T0 review · 3 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Enterprise coding agents can internalize stable schema knowledge through post-training, eliminating the need to inject exhaustive schema documentation into every prompt while improving quality, consistency, and multi-turn behavior.

desk verdict Solid engineering story on token efficiency; the headline quality gain is undercut by the judge-reward/evaluation overlap, so treat the +9.6pp as provisional. read the letter →

arxiv 2607.22642 v1 pith:CCAF4YYO submitted 2026-06-24 cs.AI cs.CLcs.LGcs.MAcs.SE

Aakash Kolekar , Sahika Genc , Shahriar Shariat , Bunyamin Sisman , Tibor Mezi , Barbara Poblete , Shree Vandana Kachroo , Calvin Chi
show 5 more authors
Parth Parmar Ari Singer Prayaas Jain Cindy Barker Benoit Dumoulin
This is my paper
classification cs.AIcs.CLcs.LGcs.MAcs.SE
keywords LargeLanguageModelsEnterpriseCodingAgentsSchema-GroundedCodeGenerationReinforcementLearningLongHorizonReasoningandToolUseSupervisedFine-TuningSchemaInternalizationMulti-TurnInteraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper investigates whether stable schema knowledge and tool-use behavior for enterprise analytics agents can be moved out of the prompt and into the model's weights. It presents CRAFT, a two-stage post-training recipe: supervised fine-tuning on schema-stripped demonstrations teaches domain-structured plans and executable code, then execution-shaped reinforcement learning aligns tool selection, plan–code consistency, and recovery from failed runs. Evaluated in a controlled advertising-analytics environment with beta APIs, CRAFT reports a +9.6 percentage-point composite Agent Score gain over a schema-stuffed baseline, a roughly 9x reduction in input tokens, and up to 5x fewer schema-discovery loops, while holding general capabilities roughly flat. A sympathetic reader would care because the default context-stuffing deployment pattern is expensive, brittle to schema changes, and weak in multi-turn use; if CRAFT holds, agents can be cheaper and more consistent without losing accuracy.

What carries the argument

The central object is the two-stage post-training recipe CRAFT, carried by three mechanisms. (1) Schema-stripped PLAN SFT: supervised fine-tuning on trajectories whose inputs omit the large DDL block and whose loss masks tool outputs, forcing the model to learn schema and API patterns as weights rather than context. (2) Execution-shaped GRPO (group-relative policy optimization): reinforcement learning with a four-term reward — outcome (execution/unit tests), process (tool-call structure, code quality), consistency (plan-code/run agreement), and judge (business alignment) — that propagates per-turn tool rewards through multi-turn rollouts. (3) Tri-Gate filtering: an execution-validity gate, a

What would settle it

Give two otherwise identical CRAFT-trained and schema-stuffed agents the same set of production-style analytical requests, and have a panel of domain experts — blind to which system produced which answer — rate final answers and follow-up quality on a business-correctness scale. If expert ratings do not show a comparable or larger gap than the internal Agent Score delta of +9.6 pp, the claimed improvement is at least partly an artifact of optimizing toward the LLM-judge rubric. Alternatively, feed both systems a schema that shares no entities with the training schema: if CRAFT's pass@1 collaps

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that a post-trained policy can internalize a proprietary analytical schema well enough that exhaustive DDL and API documentation no longer need to be injected into every prompt. The recipe strips schema context from supervised training inputs, trains the model to emit business-level plans and executable code, then applies execution-shaped reinforcement learning over a reward that combines execution correctness, tool-call structure, plan-code consistency, and judge-assessed business alignment. The reported result: relative to a schema-stuffed baseline, the composite Agent Score improves by 9.6 percentage points, consistency by 4.1 pp, and multi-turn coherenc

Load-bearing premise

The load-bearing premise is that the controlled evaluation environment — beta APIs, schema-linked data, and a rubric whose qualitative components are scored partly by LLM judges — measures the same thing real production analytics quality is, so that the reported deltas predict how the agent will behave for actual advertisers and analysts.

Editorial extensions

If this is right

  • If CRAFT is right, schema evolution no longer requires prompt-pipeline updates and regression testing for every column rename or table migration; schema knowledge lives in weights and can be refreshed in post-training.
  • Inference cost for enterprise analytics agents drops by roughly 9x in input tokens, which compounds over multi-turn conversations where schema was previously re-injected each turn.
  • Execution-shaped RL, not SFT alone, is what buys consistency and multi-turn progression: SFT alone improves quality but slightly reduces consistency, while the RL stage drives most interaction gains.
  • The same division of labor — train-internalized domain structure, align interaction policy with execution feedback — should transfer to other structured-data domains with stable schemas, such as internal database querying or financial reporting.
  • The failure taxonomy (ambiguity resolution, behavioral instability, hierarchy/aggregation errors) gives a diagnostic ordering for where post-trained analytical agents go wrong, independent of the specific evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper leaves implicit: the 9x token saving and 5x discovery-loop reduction should translate into end-to-end latency and cost gains for real users, since prompt processing and failed exploratory calls typically dominate wall-clock time; these operational benefits are implied by the efficiency ratios but not measured as latency or dollar cost.
  • The heavy reliance on judge-based reward terms for the same qualitative dimensions used in evaluation raises a testable concern the paper acknowledges only in its appendix: an independent human-SME rating study on a held-out production-like set would determine whether the +9.6 pp Agent Score reflects genuine business-facing improvement or partially optimized-for-evaluator behavior.
  • The schema-internalization hypothesis predicts a sharp failure mode: queries requiring genuinely new entities or workflows not in training will degrade, which the paper itself limits in its Limitations section; a natural stress test is measuring CRAFT's drop-off on out-of-schema queries against the schema-stuffed baseline, which should grow as schema novelty increases.
  • The Tri-Gate filter's 15% acceptance rate (15.1K accepted from 100K+) suggests a large data-engineering cost; a follow-up question is how many accepted trajectories are actually needed — whether the recipe's gains survive at 5K or 2K trajectories, which would change the economics for smaller enterprises.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes CRAFT, a two-stage post-training recipe for enterprise coding agents over proprietary advertising-analytics schemas. Stage I (PLAN SFT) trains a 120B MoE model on schema-stripped demonstrations, removing exhaustive DDL from inputs and masking tool outputs from the loss. Stage II applies GRPO with a composite reward (Eq. 3) combining execution, process, consistency, and LLM-judge terms. Training trajectories are filtered by a Tri-Gate procedure (execution validation, output completeness, LLM-judge reasoning audit) yielding 15.1K accepted trajectories. Evaluated in a beta-API enterprise sandbox on 25 entities and 30 workflows, CRAFT reports +9.6pp composite Agent Score, +4.1pp consistency, +4.2pp coherence, 0.11x input-token burden, and 0.20x schema-discovery loops relative to a schema-stuffed baseline, with no material regression on public guardrails. Claims are scoped to planned rollout and reported as confidentiality-safe deltas.

Significance. The efficiency and interaction-level results are credible and practically useful: input-token reduction is mechanical, schema-discovery-loop counts are objective, the training-stage ablation (Table 4) is informative, and the paper includes guardrail deltas and a GSM8K contamination audit. If the quality gain is real, the recipe is a meaningful contribution to deploying coding agents on stable proprietary schemas. However, the headline +9.6pp Agent Score rests on an unreleased judge-based rubric whose dimensions overlap with the RL reward (Eq. 3 vs Eq. 4). The paper itself concedes judge vulnerability (§9, App. C) but provides no human/SME validation, no γ=0 ablation, and no judge-agreement statistics. This is the central weakness; the paper's significance is currently contingent on resolving it.

major comments (3)
  1. [§4.2 Eq. (3), §5.1 Eq. (4), Table 7, Appendix C] The training reward includes γ R_judge for 'business alignment' and plan–code consistency, while the evaluation rubric's S_plan and S_ans cover the same qualitative dimensions, which Appendix C says are 'difficult to express through deterministic execution tests.' After RL the largest component gains are Plan-code (+8.4pp) and Final-answer (+6.2pp), precisely the judge-assessed dimensions. Section 9 and Appendix C concede reward-hacking vulnerability, but the paper reports no SME validation of the rubric, no γ=0 ablation, and no judge agreement or calibration statistics. The +9.6pp headline may therefore largely reflect optimization toward the evaluator. Please report a γ=0 ablation (or otherwise exclude judge terms from the reward), validate a sample of outputs with human/SME ratings, and decompose the gains into deterministic vs judge-based components.
  2. [§5.1 Eq. (4), Tables 4/7, Appendix A] The Agent Score weights (w_exec, w_tool, w_plan, w_ans) are never specified, and the component definitions are under-specified. More importantly, all quality metrics are deltas over a schema-stuffed baseline whose model, prompt construction, and DDL context are not described beyond 'same held-out tasks... decoding settings.' Since the central claim is a relative improvement over this baseline, the reader cannot interpret the magnitude or assess whether the baseline is strong or realistic. Please provide at least the baseline model and prompt details, and either disclose the rubric/weights or include de-identified example score sheets.
  3. [§3.2, Table 2] Tri-Gate's Gate 3 filters trajectories by LLM-judge reasoning audit using the same qualitative dimensions later used in the reward and evaluation. This compounds the reward-evaluation overlap: the retained 15.1K trajectories are preselected to match judge preferences before RL, so the pipeline may amplify judge bias rather than independent quality. A deterministic or human-validated subset, or an analysis of judge-filter agreement, is needed to establish that the curation step does not predetermine the reported gains.
minor comments (4)
  1. [Abstract/Table 1] 'thorough processing for Amazon Ads API definitions' should be 'through processing.'
  2. [§5.2, Table 8] General-capability guardrail deltas are reported without error bars or run counts; if these are also means over three seeded runs, the variance should be shown for consistency with other tables.
  3. [§5.3] The reproducibility statement says the authors 'intend to provide' reproduction instructions if accepted, but no concrete plan or artifact is included. This is understandable given confidentiality, but the current manuscript should state more explicitly what will and will not be released at review time.
  4. [§2 and throughout] The term 'beta APIs' is used throughout but never defined. A sentence clarifying how beta APIs differ from the expected production tool surface would help readers assess the evaluation environment's representativeness.

Circularity Check

1 steps flagged · score 6.0 of 10

RL reward and Agent Score evaluation share an LLM-judge rubric for plan-code alignment; part of the headline gain reduces to optimizing the evaluator.

  1. fitted input called prediction [Section 4.2 (Eq. 3, Table 3), Section 5.1 (Eq. 4), Appendix C]
    "Eq. (3): "R_total = R_outcome + α R_process + λ R_consistency + γ R_judge"; Table 3: "R_judge ... LLM-judge assessment ... Business alignment"; Eq. (4): "AgentScore = w_exec S_exec + w_tool S_tool + w_plan S_plan + w_ans S_ans ... S_plan measures alignment between the stated analytical plan and generated code"; Sec. 5.1: "All systems use the same ... judge rubrics"; Appendix C: "judge-based signals only for dimensions that are difficult to express through deterministic execution tests, such as business alignment and plan–code consistency.""

    The RL objective includes γR_judge, an LLM-judge reward for business alignment and plan–code consistency. The Agent Score used to report the headline +9.6 pp gain includes S_plan, which is defined as plan–code alignment — the same dimension Appendix C says is judge-assessed because deterministic tests are insufficient. Section 5.1 says the systems share the same judge rubrics, so the policy is rewarded by a judge evaluating the same construct that later certifies quality. The +8.4 pp plan-code component (Table 7) therefore partially reduces to optimizing the evaluator. The paper's own Limitations concedes "LLM-judge feedback may be vulnerable to reward hacking or stylistic over-optimization," and no γ=0 ablation or human/SME rubric validation is reported.

full rationale

The central claim that CRAFT improves coding-agent quality is partially circular: Eq. (3) optimizes an LLM-judge reward (R_judge) for business alignment and plan–code consistency, while Eq. (4)'s Agent Score — the primary quality measure — includes S_plan for plan–code alignment, which Appendix C identifies as judge-assessed. Because training and evaluation share judge rubrics, the judge-scored components of the +9.6 pp gain are not independent of the training signal. This is not full circularity: execution correctness, tool-use validity, the ~9x token reduction, the 5x reduction in schema-discovery loops, and external guardrail benchmarks (IFEval, GSM8K, GPQA) are deterministic or externally grounded, and the efficiency gains follow by construction from schema stripping rather than by reward-evaluator overlap. The self-citations [20] and [26] appear only in related work and are not load-bearing; no uniqueness theorem is imported. The paper's Limitations section itself flags judge-based reward-hacking risk, supporting the partial-circularity reading. Score 6 reflects one substantive evaluator-overlap step rather than a fully forced derivation.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

CRAFT does not introduce new physical or formal entities; its central claim rests on the representativeness of the controlled beta-API environment, the coverage quality of the curated 15.1K trajectories, and the validity of judge-based measurements. The undisclosed reward and metric weights make the reported composite improvements sensitive to choices the reader cannot inspect.

free parameters (3)
  • Reward weights alpha, lambda, gamma = undisclosed
    Eq. (3) combines outcome, process, consistency, and judge rewards with weights that are never reported; the relative contribution of each signal to the +9.6pp gain is therefore unidentified.
  • Agent Score weights w_exec, w_tool, w_plan, w_ans = undisclosed
    Eq. (4) defines the composite Agent Score with unspecified weights; the headline aggregate delta cannot be independently reconstructed from the Table 7 component deltas without these weights.
  • KL penalty coefficient beta = undisclosed
    Eq. (2) includes beta KL divergence against the reference policy; the value controls how far the policy can drift and is not reported.
assumptions (6)
  • domain assumption The controlled beta-API environment and schema-linked data faithfully represent production advertising analytics.
    Section 2 states the evaluation uses beta APIs and controlled data; if these diverge from the real production surface in schema, data distribution, or API behavior, the reported deltas may not transfer to rollout.
  • domain assumption The 15.1K accepted trajectories provide sufficient coverage of the 25 entities and 30 agentic workflows.
    Table 2 reports the filtering counts but no coverage analysis over prompt complexity, entity combinations, or workflow families; under-coverage would limit the learned schema internalization.
  • domain assumption LLM-judge assessments are a valid proxy for human/SME judgments in this domain.
    Gate 3 and R_judge rely on LLM-judge reasoning audits; the paper cites general LLM-judge literature but reports no in-domain human correlation or inter-annotator agreement study.
  • domain assumption Execution success and data-integrity checks are sufficient correctness signals for training and evaluation.
    Sections 3.2 and 4.2 treat execution as the primary grounding signal; this assumes beta-API execution outcomes correspond to business-facing correctness, including edge cases like null handling and ambiguity resolution.
  • domain assumption The target schema is stable over the deployment horizon.
    Section 9 explicitly bounds CRAFT to stable enterprise schemas; if schemas change frequently, the schema-stripping benefit requires retraining or retrieval augmentation and the reported gains may not hold.
  • standard math GRPO and the standard RL objective assumptions hold for multi-turn tool-use trajectories.
    The paper uses GRPO directly (Eq. 2) with no new objective derivation; it assumes the estimator, advantage computation, and KL penalty behave correctly in the extended multi-turn rollout setting described in Appendix D.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CRAFT: Learn the Schema, Execute the Plan." pith.science (2026). https://pith.science/paper/CCAF4YYO

@misc{pith2026260722642,
  author       = {Pith},
  title        = {Pith review of: CRAFT: Learn the Schema, Execute the Plan},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CCAF4YYO}},
  note         = {Machine review of arXiv:2607.22642}
}
read the original abstract

Enterprise coding agents translate natural-language analytical requests into executable code over proprietary APIs, schemas, and metric definitions. Yet the prevailing deployment pattern injecting exhaustive schema and tool documentation into each prompt increases inference overhead, complicates schema evolution, and undermines reliability in multi-turn analysis. We investigate whether stable schema knowledge and tool-use behavior can instead be acquired through post-training while preserving the consistency required for production-facing analytics. We present CRAFT, a two-stage post-training recipe for schema-grounded coding agents. First, schema-stripped PLAN supervised fine-tuning learns domain-structured plans and executable behaviors from validated trajectories without exhaustive prompt-time schema injection. Second, execution-shaped reinforcement learning aligns the policy for tool selection, code quality, plan-code consistency, and recovery from failed executions. Training trajectories are curated through a Tri-Gate filter combining execution validation, data-integrity checks, and LLM-judge reasoning audit. We evaluate CRAFT for planned rollout in advertising analytics, covering campaign performance analysis, metric drill-downs, entity-level performance analysis, and multi-turn analytical refinement. The enterprise evaluation environment incorporates beta APIs as the agent-facing tool surface and spans 25 schema-linked core entities and 30 agentic workflows. Relative to a schema-stuffed baseline, CRAFT improves composite Agent Score by +9.6 pp, consistency by +4.1 pp, and multi-turn coherence by +4.2 pp, while reducing input-token burden by approximately 9x and schema-discovery loops by up to 5x. We further report deployment tradeoffs, reward-shaping limitations, and training-infrastructure extensions required for multi-turn tool-use reinforcement learning in enterprise settings.

Figures

Figures reproduced from arXiv: 2607.22642 by the authors.

Figure 1
Figure 1. CRAFT system overview. In the existing diagram, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 1 canonical work pages

  1. [1]

    Amazon Ads. 2026. Amazon Ads Advanced Tools Center Public API v1. https: //advertising.amazon.com/API/docs/en-us/amazon-ads/1-0/apis. Accessed: 2026- 05-23

    2026

  2. [2]

    Amazon Ads. 2026. Amazon Ads API overview. https://advertising.amazon.com/ API/docs/en-us/guides/overview. Accessed: 2026-05-23

    2026

  3. [3]

    Jiaju Chen, Yuxuan Lu, Xiaojie Wang, Huimin Zeng, Jing Huang, Jiri Gesi, Ying Xu, Bingsheng Yao, and Dakuo Wang. 2025. Multi-Agent-as-Judge: Aligning LLM- Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation. arXiv preprint arXiv:2507.21028(2025). doi:10.48550/arxiv.2507.21028

    arXiv doi:10.48550/arxiv.2507.21028 2025

  4. [4]

    Daya Guo, Dejian Yang, et al . 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.arXiv preprint arXiv:2501.12948 (2025)

    arXiv 2025

  5. [5]

    Junqing He, Kunhao Pan, Xiaoqun Dong, Zhuoyang Song, LiuYiBo LiuYiBo, Qianguosun Qianguosun, Yuxin Liang, Hao Wang, Enming Zhang, and Jiaxing Zhang. 2024. Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Training.Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistic...

    doi:10.18653/v1/2024.acl-long.736 2024

  6. [6]

    Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisenschlos. 2020. TAPAS: Weakly Supervised Table Parsing via Pre-training. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 4320–4333

    2020

  7. [7]

    Seungone Kim, Jamin Shin Suk, Shayne Longpre, Bill Yuchen Zhou, Jamin Kim, James Kim, Seonghyeon Lee, Seung-won Yun, Minjoon Shin, and Minjoon Seo

  8. [8]

    Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. 2025. Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows. InThe Thirteenth International Conference on Learning Repr...

    2025

Show all 34 references
  1. [9]

    uttler, Mike Lewis, Wen-tau Yih, Tim Rockt

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K"uttler, Mike Lewis, Wen-tau Yih, Tim Rockt"aschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. InAdvances in...

    2020

  2. [10]

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al . 2023. BIRD: A Big Bench for Large-Scale Database Grounded Text-to-SQL Evaluation.arXiv preprint arXiv:2305.03111(2023)

    2023 arXiv

  3. [11]

    Minghao Li, Feifan Zhao, Xing Li, Han Yu, Shizhu Su, Hang Xu, Jie Wu, and Ling- peng Kong. 2023. API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs.arXiv preprint arXiv:2304.08244(2023)

    2023 arXiv

  4. [12]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics 12 (2024), 277–294

    2024

  5. [13]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kai Men, Kejuan Yang, et al . 2023. AgentBench: Evaluating LLMs as Agents.arXiv preprint arXiv:2308.03688(2023)

    2023 arXiv

  6. [14]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Go- rilla: Large Language Model Connected with Massive APIs.arXiv preprint arXiv:2305.15334(2023). doi:10.48550/arxiv.2305.15334 Kolekar et al

    2023 doi arXiv

  7. [15]

    Mohammadreza Pourreza and Davood Rafiei. 2024. DIN-SQL: Decomposed In- Context Learning of Text-to-SQL with Self-Correction. InAdvances in Neural Information Processing Systems, Vol. 36

    2024

  8. [16]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al . 2024. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-World APIs. InThe Twelfth International Conference on Learning Representations

    2024

  9. [17]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. Toolformer: Lan- guage models can teach themselves to use tools.Advances in Neural Information Processing Systems36 (2024)

    2024

  10. [18]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Feng, Mingchuan Fang, Yuxiang Wu, Xiaoqiang Liu, et al. 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300(2024)

    2024 arXiv

  11. [19]

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning.arXiv preprint arXiv:2303.11366(2023). doi:10.48550/ arxiv.2303.11366

    2023 doi arXiv

  12. [20]

    Andy Sun, Tianqi Zheng, Aakash Kolekar, Rohit Patki, Hossein Khazaei, Xuan Guo, George Cai, David Liu, Ruirui Li, Yupin Huang, Dante Everaert, Hanqing Lu, Garima Patel, and Monica Cheng. 2024. A product-aware query auto-completion framework for e-commerce search via retrieval-...

    2024

  13. [21]

    THUDM. 2025. slime: An LLM Post-Training Framework for RL Scaling. https: //github.com/THUDM/slime. Accessed: 2026-05-23

    2025

  14. [22]

    verl-project. 2025. verl: Volcano Engine Reinforcement Learning for LLMs. https://github.com/verl-project/verl. Accessed: 2026-05-23

    2025

  15. [23]

    Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. Rat-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 7567–7578. doi:10.1...

    2020 doi

  16. [24]

    Huaijie Wang, Shibo Hao, Hanze Dong, Shenao Zhang, Yilin Bao, Ziran Yang, and Yi Wu. 2024. Offline Reinforcement Learning for LLM Multi-Step Reasoning. arXiv:2412.16145 [cs.LG] https://arxiv.org/abs/2412.16145

    2024 arXiv

  17. [25]

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024. Executable Code Actions Elicit Better LLM Agents. InForty-first International Conference on Machine Learning

    2024

  18. [26]

    Yuchen Yan, Aakash Kolekar, Sahika Genc, Wenju Xu, Edward W Huang, Anirudh Srinivasan, Mukesh Jain, Qi He, and Hanghang Tong. 2025. To Answer or Not to Answer (TAONA): A Robust Textual Graph Understanding and Question Answering Approach. InFindings of the Association for Compu...

    2025 doi

  19. [27]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. ReAct: Synergizing Reasoning and Acting in Language Models.arXiv preprint arXiv:2210.03629(2022). doi:10.48550/arxiv.2210.03629

    2022 doi arXiv

  20. [28]

    Hiroshi Yoshihara, Taiki Yamaguchi, and Yuichi Inoue. 2025. A Practical Two-Stage Recipe for Mathematical LLMs: Maximizing Accuracy with SFT and Efficiency with Reinforcement Learning. arXiv:2507.08267 [cs.LG] https: //arxiv.org/abs/2507.08267

    2025 arXiv

  21. [29]

    Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard So...

  22. [30]

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. InPr...

    2018 doi

  23. [31]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT- Bench and Chatbot Arena. InAdvances in Neural Information P...

    2023

  24. [32]

    Ruiqi Zhong, Tao Yu, and Dan Klein. 2020. Semantic Evaluation for Text-to-SQL with Distilled Test Suites. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 396–411. doi:10.18653/v1/2020.emnlp- main.29 GenAI Usage Disclosure The authors u...

    2020 doi

  25. [2019]

    InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing

    CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. 1962...

    2019 doi

  26. [2023]

    Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models.arXiv preprint arXiv:2310.08491(2023)

    2023 arXiv

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.