Pith. sign in

REVIEW 10 cited by

FullStack Bench: Evaluating LLMs as Full Stack Coders

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.00535 v6 pith:4YORXJVV submitted 2024-11-30 cs.AI cs.SE

Bytedance-Seed-Foundation-Code-Team: Yao Cheng , Jianfeng Chen , Jie Chen , Li Chen , Liyu Chen , Wentao Chen , Zhengyu Chen , Shijie Geng
show 46 more authors
Aoyan Li Bo Li Bowen Li Linyi Li Boyi Liu Jiaheng Liu Kaibo Liu Qi Liu Shukai Liu Siyao Liu Tianyi Liu Tingkai Liu Yongfei Liu Rui Long Jing Mai Guanghan Ning Z.Y. Peng Kai Shen Jiahao Su Jing Su Tao Sun Yifan Sun Yunzhe Tao Guoyin Wang Siwei Wang Xuwu Wang Yite Wang Zihan Wang Jinxiang Xia Liang Xiang Xia Xiao Yongsheng Xiao Chenguang Xi Shulin Xin Jingjing Xu Shikun Xu Hongxia Yang Jack Yang Yingxiang Yang Jianbo Yuan Jun Zhang Yufeng Zhang Yuyu Zhang Shen Zheng He Zhu Ming Zhu
This is my paper
classification cs.AIcs.SE
keywords benchfullstackprogrammingcodedomainsapplicationcapabilitiescomprehensive
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

As the capabilities of code large language models (LLMs) continue to expand, their applications across diverse code intelligence domains are rapidly increasing. However, most existing datasets only evaluate limited application domains. To address this gap, we have developed a comprehensive code evaluation dataset FullStack Bench focusing on full-stack programming, which encompasses a wide range of application domains (e.g., basic programming, data analysis, software engineering, mathematics, and machine learning). Besides, to assess multilingual programming capabilities, in FullStack Bench, we design real-world instructions and corresponding unit test cases from 16 widely-used programming languages to reflect real-world usage scenarios rather than simple translations. Moreover, we also release an effective code sandbox execution tool (i.e., SandboxFusion) supporting various programming languages and packages to evaluate the performance of our FullStack Bench efficiently. Comprehensive experimental results on our FullStack Bench demonstrate the necessity and effectiveness of our FullStack Bench and SandboxFusion.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CodeContests+: High-Quality Test Case Generation for Competitive Programming

    cs.SE 2025-06 conditional novelty 7.0 of 10

    A new dataset, CodeContests+, uses an LLM agent system to generate and validate competitive programming test cases, improving evaluation accuracy and reinforcement learning training.

  2. Towards Reliable C-to-Rust Translation with Rule-Guided Reasoning and Reinforcement Learning

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A rule-guided MCTS plus dual-reward reinforcement learning pipeline improves LLM-based C-to-Rust translation accuracy and cuts unsafe Rust output across three benchmarks.

  3. Vibration-Based Energy Metric for Restoring Needle Alignment in Autonomous Robotic Ultrasound

    cs.RO 2025-08 unverdicted novelty 6.0 of 10

    A vibration-based energy metric, obtained by periodically vibrating the needle, is used as an out-of-plane feedback signal to re-align the ultrasound probe with the needle plane, reporting 0.41 mm translation and 0.51...

  4. IFEvalCode: Controlled Code Generation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A 1,620-sample, 8-language, Chinese/English benchmark separates code correctness from instruction-following and shows instruction compliance is far lower than correctness across 40+ LLMs.

  5. Turning the Tide: Repository-based Code Reflection

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A new 1,888-case, six-language benchmark and a trained model measure and improve how code LLMs edit and repair code inside multi-file repositories.

  6. AdaptiveLLM: A Framework for Selecting Optimal Cost-Efficient LLM for Code-Generation Based on CoT Length

    cs.SE 2025-06 conditional novelty 6.0 of 10

    AdaptiveLLM routes coding problems to cost-effective LLMs by clustering chain-of-thought lengths into three difficulty levels, improving pass@1 by 7.86% over ComplexityNet while cutting inference cost by 88.9%.

  7. Seed-Coder: Let the Code Model Curate Data for Itself

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Seed-Coder shows that an LLM-trained quality scorer can filter 6T tokens of code data and yield 8B models that outperform similar-size open code models.

  8. SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD

    cs.CL 2026-07 conditional novelty 5.0 of 10

    An Ascend-NPU training stack reaches 34.22% MFU on DeepSeek-V4-Pro, and a solver-verified CPT+SFT recipe raises OR benchmark averages to 71.81% (Flash) and 77.33% (Pro).

  9. UniCode: Augmenting Evaluation for Code Reasoning

    cs.SE 2025-10 conditional novelty 5.0 of 10

    UniCode's LLM-generated coding benchmark drops top-model pass@1 to 70.3% and indicates current LLMs rely on memorized seed logic instead of generalizing to new algorithmic problems.

  10. Dream-Coder 7B: An Open Diffusion Language Model for Code

    cs.CL 2025-09 conditional novelty 5.0 of 10

    An open 7B diffusion language model for code, adapted from Qwen2.5-Coder, reaches 21.4% pass@1 on LiveCodeBench and shows adaptive, non-left-to-right generation patterns.

Pith tools