REVIEW 16 cited by
OctoPack: Instruction Tuning Code Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Finetuning large language models (LLMs) on instructions leads to vast performance improvements on natural language tasks. We apply instruction tuning using code, leveraging the natural structure of Git commits, which pair code changes with human instructions. We compile CommitPack: 4 terabytes of Git commits across 350 programming languages. We benchmark CommitPack against other natural and synthetic code instructions (xP3x, Self-Instruct, OASST) on the 16B parameter StarCoder model, and achieve state-of-the-art performance among models not trained on OpenAI outputs, on the HumanEval Python benchmark (46.2% pass@1). We further introduce HumanEvalPack, expanding the HumanEval benchmark to a total of 3 coding tasks (Code Repair, Code Explanation, Code Synthesis) across 6 languages (Python, JavaScript, Java, Go, C++, Rust). Our models, OctoCoder and OctoGeeX, achieve the best performance across HumanEvalPack among all permissive models, demonstrating CommitPack's benefits in generalizing to a wider set of languages and natural coding tasks. Code, models and data are freely available at https://github.com/bigcode-project/octopack.
Forward citations
Cited by 16 Pith papers
-
DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search
With matched open data and backbones, ColBERT-style late interaction turns English translate-train into multilingual generalization, while dense retrieval stays mostly inside the translated languages.
-
SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
SPEED-Bench is a new standardized benchmark for speculative decoding that supplies semantically diverse qualitative data and throughput-oriented splits across concurrency levels, integrated with vLLM and TensorRT-LLM.
-
Rethinking Code Complexity Through the Lens of Large Language Models
LM-CC, a code complexity metric built from an LLM's token entropy and a semantic compositional hierarchy, correlates with LLM task performance (r ≈ −0.92…−0.97 after length control) where classical metrics mostly fail.
-
AI-Assisted Fixes to Code Review Comments at Scale
Fine-tuned Llama models generate exact-match patches for 68% of internal code review comments, and a safety trial shows AI suggestions slow reviewers unless hidden from them.
-
Multilingual Multimodal Software Developer for Code Generation
A 7B vision-language model trained on synthetic diagram-to-code data outperforms several larger open-weight models on a new 10-language UML/flowchart code-generation benchmark.
-
Recommending Variable Names for Extract Local Variable Refactorings
VarNamer recommends variable names for extract local variable refactorings using homogeneous-variable reuse and mined naming rules, improving exact-match precision over Eclipse, IntelliJ, and Incoder.
-
Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization
Reinforcement learning with execution feedback enables a code model to iteratively improve the efficiency of its own generated code, surpassing supervised and preference-based training methods.
-
ExeSQL: Self-Taught Text-to-SQL Models with Execution-Driven Bootstrapping for SQL Dialects
Execution-driven bootstrapping, where a model generates SQL, executes it, and keeps only queries that run, lets a 7B model outperform GPT-4o on PostgreSQL, MySQL, and Oracle text-to-SQL benchmarks.
-
Using Mutation-Analysis to Examine an LLM's Ability to Summarize Code
Mutation-based testing shows LLM-generated code summaries frequently fail to reflect behavior-changing edits, with accuracy dropping as code complexity rises.
-
AutoMixAlign: Adaptive Data Mixing for Multi-Task Preference Optimization in LLMs
AutoMixAlign adaptively reweights or resamples task data during DPO training to match specialist-model losses, improving average performance on helpfulness, coding, and safety benchmarks compared to standard DPO and m...
-
Lossless Token Sequence Compression via Meta-Tokens
A new compression scheme replaces repeated token subsequences with learnable placeholder tokens, shrinking prompts by 15-27% with no loss of information, and fine-tuned LLMs perform nearly as well as on uncompressed input.
-
PhotoArtAgent: Intelligent Photo Retouching with Language Model-Based Artist Agents
A training-free LLM agent that edits photos through Lightroom with an analyze-apply-reflect loop scored 6.50 out of 10 in a user study, ahead of automated baselines and two human experts.
-
Deep Research Agents: A Systematic Examination And Roadmap
A survey that organizes LLM-powered deep research agents into static versus dynamic workflows and single versus multi agent architectures, and reviews their benchmarks and open challenges.
-
VulBinLLM: LLM-powered Vulnerability Detection for Stripped Binaries
An LLM pipeline that enriches decompiled code with vulnerability hints and then classifies CWEs reports high Juliet accuracy, but leaky hints and unmatched baselines weaken the claim.
-
Position: Agentic Systems Constitute a Key Component of Next-Generation Intelligent Image Processing
Image processing should move from monolithic deep models to agentic systems that orchestrate multiple tools, with a proposed six-level autonomy ladder.
-
Applying the Chinese Wall Reverse Engineering Technique to Large Language Model Code Editing
Using Gemini 2.5 Pro to annotate code with edit instructions improved Comma v0.1 1T's CanItEdit pass@20 from 20.00 to 33.33 and Starcoder2 Instruct's pass@1 from 35.10 to 42.05.
Discussion (0). Continue with ORCID to comment.