Pith. sign in

REVIEW 4 cited by

CodeShell Technical Report

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.15747 v1 pith:IEXJWL5P submitted 2024-03-23 cs.SE cs.AI

Rui Xie , Zhengran Zeng , Zhuohao Yu , Chang Gao , Shikun Zhang , Wei Ye This is my paper · ORCID
classification cs.SEcs.AI
keywords datacodecodeshell-basebillioncodellamacomprehensionfilteringhigh-quality
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Code large language models mark a pivotal breakthrough in artificial intelligence. They are specifically crafted to understand and generate programming languages, significantly boosting the efficiency of coding development workflows. In this technical report, we present CodeShell-Base, a seven billion-parameter foundation model with 8K context length, showcasing exceptional proficiency in code comprehension. By incorporating Grouped-Query Attention and Rotary Positional Embedding into GPT-2, CodeShell-Base integrates the structural merits of StarCoder and CodeLlama and forms its unique architectural design. We then carefully built a comprehensive data pre-processing process, including similar data deduplication, perplexity-based data filtering, and model-based data filtering. Through this process, We have curated 100 billion high-quality pre-training data from GitHub. Benefiting from the high-quality data, CodeShell-Base outperforms CodeLlama in Humaneval after training on just 500 billion tokens (5 epochs). We have conducted extensive experiments across multiple language datasets, including Python, Java, and C++, and the results indicate that our model possesses robust foundational capabilities in code comprehension and generation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench

    cs.SE 2025-06 conditional novelty 6.0 of 10

    UTBoost adds LLM-generated tests and a repaired log parser to SWE-Bench, uncovering 345 falsely-passing patches and causing dozens of leaderboard ranking shifts.

  2. LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks

    cs.SE 2025-02 conditional novelty 6.0 of 10

    Across 83 SE benchmarks, average leakage into StarCoder's pretraining data is 4.8% (Python), 2.8% (Java), and 0.7% (C/C++), but QuixBugs and BigCloneBench are 100% and 55.7% leaked.

  3. Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A tree-based, inference-only framework combining execution metrics with LLM self-critique improves code generation correctness and efficiency across models and benchmarks.

  4. Can You Really Trust Code Copilots? Evaluating Large Language Models from a Code Security Perspective

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A new four-task benchmark and a fine-tuned judge model show that leading AI coding assistants still generate vulnerable code, even though many can spot vulnerabilities when prompted.

Pith tools