Pith. sign in

REVIEW 20 cited by

SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2503.15478 v1 pith:Y45YWD6T submitted 2025-03-19 cs.LG

Yifei Zhou , Song Jiang , Yuandong Tian , Jason Weston , Sergey Levine , Sainbayar Sukhbaatar , Xian Li This is my paper · ORCID
classification cs.LG
keywords multi-turnagentsalgorithmsmodelsweet-rltasksbenchmarkcolbench
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language model (LLM) agents need to perform multi-turn interactions in real-world tasks. However, existing multi-turn RL algorithms for optimizing LLM agents fail to perform effective credit assignment over multiple turns while leveraging the generalization capabilities of LLMs and it remains unclear how to develop such algorithms. To study this, we first introduce a new benchmark, ColBench, where an LLM agent interacts with a human collaborator over multiple turns to solve realistic tasks in backend programming and frontend design. Building on this benchmark, we propose a novel RL algorithm, SWEET-RL (RL with Step-WisE Evaluation from Training-time information), that uses a carefully designed optimization objective to train a critic model with access to additional training-time information. The critic provides step-level rewards for improving the policy model. Our experiments demonstrate that SWEET-RL achieves a 6% absolute improvement in success and win rates on ColBench compared to other state-of-the-art multi-turn RL algorithms, enabling Llama-3.1-8B to match or exceed the performance of GPT4-o in realistic collaborative content creation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 20 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CustomerSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    SalesSim benchmarks MLLMs as retail user simulators, finds gaps in persona adherence and over-persuasion, and introduces UserGRPO RL to raise decision alignment by 13.8%.

  2. TimeHC-RL: Temporal-aware Hierarchical Cognitive Reinforcement Learning for Enhancing LLMs' Social Intelligence

    cs.CL 2025-05 conditional novelty 7.0 of 10

    A temporal-aware hierarchical reinforcement learning method improves a 7B LLM's social reasoning enough to rival DeepSeek-R1 and OpenAI-O3 on in-domain theory-of-mind benchmarks.

  3. TCPO: Turn-Level Credit Policy Optimization

    cs.AI 2026-08 conditional novelty 6.0 of 10

    TCPO improves multi-turn LLM RL by converting verifier scores into turn-level credit via three reference-based comparisons, beating baseline methods on math, code, and agent tasks.

  4. Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning

    cs.LG 2026-07 reject novelty 6.0 of 10

    MetaEvolve trains LLMs with reinforcement learning on synthesized code-refinement trajectories, reporting large gains on coding and numerical-optimization benchmarks.

  5. AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use

    cs.AI 2026-07 conditional novelty 6.0 of 10

    AppWorld-UL perturbs AppWorld's autonomous tasks into 516 user-interaction-requiring tasks; SOTA agents succeed on only 48.6% of them.

  6. PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    PAIR combines a hidden-state probe with an attention correction to deliver robust step-level rewards for GRPO-based optimization of multi-turn LLM agents, achieving high AUROC on contaminated trajectories at low cost.

  7. ABBEL: Learning Natural-Language Belief States for Memory-Efficient Interaction

    cs.CL 2025-12 conditional novelty 6.0 of 10

    ABBEL trains LLM agents to act from a compact natural-language belief state; belief-quality and brevity rewards close most of the gap with full-context agents.

  8. Conversation Forests: The Key to Fine Tuning Large Language Models for Multi-Turn Medical Conversations is Branching

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Branched multi-turn conversation training outperformed linear training on simulated medical interviews for Llama-3.1-8B and Ministral-8B.

  9. PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A new multi-turn reinforcement learning framework trains a single LLM to both solve math problems and verify its own solutions, revising only when its verifier finds a mistake.

  10. Self-Challenging Language Model Agents

    cs.AI 2025-06 conditional novelty 6.0 of 10

    A language model agent can generate its own verifiable training tasks and improve its tool-use success rate by about 2x without human-annotated data.

  11. ARIA: Training Language Agents with Intention-Driven Reward Aggregation

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Clustering language-agent actions into shared intentions and averaging their rewards reduces reward variance and improves policy performance in open-ended dialogue tasks.

  12. lmgame-Bench: How Good are LLMs at Playing Games?

    cs.AI 2025-05 conditional novelty 6.0 of 10

    lmgame-Bench turns six classic games into a scaffolded LLM evaluation suite, ranks 13 models, detects contamination, and reports RL transfer from Sokoban or Tetris to unseen games and planning tasks.

  13. How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMs

    cs.MA 2025-07 conditional novelty 5.0 of 10

    A leader LLM trained with a GRPO variant that conditions on frozen agent responses improves both collaborative and zero-shot accuracy on BBH, MATH, and MMLU.

  14. OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation

    cs.AI 2025-05 conditional novelty 5.0 of 10

    A hierarchical multi-agent framework with a domain-agnostic planner trained by SFT and DPO achieves 69.70% on GAIA and lifts a Qwen2.5-32B planner by 16.37 points.

  15. ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay

    cs.CV 2025-05 conditional novelty 5.0 of 10

    ARPO combines GRPO reinforcement learning with a replay buffer of successful GUI trajectories and task filtering, improving UI-Tars to 29.9% on OSWorld.

  16. ODYSSE: Episode-wise Policy Optimization for Personalized Agentic Reasoning

    cs.AI 2026-07 conditional novelty 4.0 of 10

    An episode-level variant of GRPO (ESPO) improves personalized GUI-agent reasoning on the 102-episode SmartSpot benchmark, outperforming step-wise and outcome-only training baselines.

  17. General Modular Harness for LLM Agents in Multi-Turn Gaming Environments

    cs.AI 2025-07 conditional novelty 4.0 of 10

    A modular perception-memory-reasoning harness consistently improves LLM game scores across four puzzle games, with perception helping spatial games and memory helping long-horizon games.

  18. SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution

    cs.CL 2025-05 conditional novelty 4.0 of 10

    SPA-RL attributes a single final task reward to individual agent steps and uses the attributions plus an executability signal as dense rewards for PPO, yielding modest success-rate gains on WebShop, ALFWorld, and VirtualHome.

  19. Get Experience from Practice: LLM Agents with Record & Replay

    cs.LG 2025-05 reject novelty 4.0 of 10

    AgentRR is a proposed paradigm that records agent traces, generalizes them into multi-level experiences, and replays them under safety checks to make LLM agents cheaper, faster, and more reliable.

  20. Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective

    cs.LG 2025-06 conditional novelty 3.0 of 10

    Under the assumption that the response reward equals the discounted sum of token rewards, response-level rewards suffice for unbiased token-level policy gradients in LLMs.

Pith tools