Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

Efficient Code Embeddings from Code Generation Models

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that 0.5B and 1.5B code-generation language models, adapted with last-token pooling, task-specific instruction prefixes, and contrastive training, match or exceed much larger embedding models on code retrieval benchmarks.

desk verdict Useful recipe for small code embedding models, but the unqualified SOTA claim doesn't hold up and the missing decontamination statement makes the headline numbers unverifiable. read the letter →

arxiv 2508.21290 v1 pith:Q2QABSZP submitted 2025-08-29 cs.CL cs.AIcs.IR

Daria Kryvosheieva , Saba Sturua , Michael Günther , Scott Martens , Han Xiao This is my paper
classification cs.CLcs.AIcs.IR
keywords codeembeddingsretrievallast-tokenpoolinginstructionprefixescontrastivelearningInfoNCElossMatryoshkarepresentationQwen2.5-Coder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that small code-generation language models—0.5B and 1.5B parameters—can be turned into code embedding models that rival or exceed much larger, more expensive systems. The recipe is to take a code LLM pre-trained on text and code, prefix queries and documents with short task-specific instructions, pool the last token's hidden state into an embedding, and fine-tune with a contrastive loss plus dimension-truncatable Matryoshka training. The resulting models beat a similar-size general-purpose embedding model and, on average, beat much larger general-purpose embedding models across 25 code retrieval benchmarks. If the claim holds, it means cheap, small models can power code retrieval for AI coding assistants and retrieval-augmented generation without needing multi-billion-parameter embedding models. The paper's main unstated assumption is that the evaluation splits were not seen during training, since training data includes splits of the same code retrieval benchmarks used in evaluation.

What carries the argument

The central mechanism is last-token pooling on a decoder-only code-generation backbone combined with task-specific instruction prefixes: each query and document is prefixed with a short task instruction (for example, 'Find the most relevant code snippet given the following query:' for a query and 'Candidate code snippet:' for a document), and the embedding is the hidden state of the final token after the prefixed text is processed. The five-way task taxonomy (NL2Code, TechQA, Code2Code, Code2NL, Code2Completion) organizes the training data, and the InfoNCE contrastive loss with a temperature of 0.05 and Matryoshka representation learning produce truncated, normalized embeddings that pull mat

What would settle it

Re-run the evaluation on a code-retrieval benchmark whose test set was published after training data collection and is verifiably absent from every training corpus in the paper's Table 3. If the average performance advantage over the comparison models disappears or shrinks sharply, the state-of-the-art claim rests on training/evaluation overlap. A simpler check: exact-match or fuzzy-match the queries and documents of CoSQA+ and the code retrieval test splits against the training datasets listed in Table 3.

Watch

Extended reading notes

Core claim

The central claim is that an autoregressive code-generating LLM can be adapted directly into a state-of-the-art code embedding model. The paper builds jina-code-embeddings-0.5b and 1.5b from Qwen2.5-Coder backbones by using the final hidden layer of the last token as the embedding, prefixing each input with task-specific instruction strings (separate strings for queries and documents across five task categories: natural-language-to-code, technical QA, code-to-code, code-to-natural-language, and code-to-completion), and training with InfoNCE contrastive loss and Matryoshka representation learning. The paper reports that both models outperform the similar-size Qwen3-Embedding-0.6B on most benc

Load-bearing premise

The load-bearing assumption is that the reported benchmark numbers measure generalization rather than memorization: the paper trains on datasets that include splits of the same code retrieval benchmarks used in evaluation, and never states that the evaluation sets were held out during training.

Editorial extensions

If this is right

  • Code retrieval can run on models under 1.5B parameters, lowering the compute and memory cost of embedding-based retrieval in AI coding tools.
  • The construction recipe—pretrained code generator plus last-token pooling plus instruction prefixes plus contrastive training—can be applied to other code LLMs, including larger or newer ones, likely improving code embeddings further.
  • Matryoshka training means users can choose lower embedding dimensions for storage- or speed-constrained deployments with a predictable trade-off in retrieval quality.
  • The five task categories give a reusable labeling scheme for future code embedding training data, separating NL2Code, TechQA, Code2Code, Code2NL, and Code2Completion.
  • If the claim holds, specialized small embedding models become an attractive default for code retrieval in retrieval-augmented generation pipelines over much larger general-purpose embedding models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the training set includes splits of the same code retrieval benchmarks used in evaluation, the reported averages depend on the unstated assumption that test splits were excluded from training; a held-out evaluation would be needed to rule out leakage inflating the results.
  • The pooling ablation shows only small differences among last-token, mean, and latent attention pooling for the same data, so the instruction prefixes and backbone choice may be carrying most of the performance; ablating prefixes independently would test this.
  • The recipe should transfer to non-English and multilingual code queries, since the synthetic multilingual CodeChef data suggests crosslingual code-to-code search works, but the paper only tests English instruction strings; extending the instruction set is a direct next test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents jina-code-embeddings-0.5b and jina-code-embeddings-1.5b, two decoder-only code embedding models built from Qwen2.5-Coder backbones. The proposed recipe uses last-token pooling, task-specific instruction prefixes for five code-retrieval task types, and contrastive InfoNCE training (optionally with Matryoshka representation learning) on a mix of public, adapted, and GPT-4o-synthesized query–document pairs. The authors evaluate on the MTEB-CoIR benchmark plus several code-related MTEB tasks, CoSQA+, and in-house benchmarks, and claim state-of-the-art performance despite the small model sizes. The empirical core of the paper is the comparison in Table 2.

Significance. If the reported results are valid, the paper would make a useful contribution: it demonstrates that compact code-generation LLMs can be adapted into general-purpose code embedding models with competitive accuracy, and the task-prefix taxonomy plus the pooling ablation provide practical guidance. The paper is also transparent about training data provenance and hyperparameters. However, the central claim of state-of-the-art performance is currently not supported by the evidence as presented: (i) the training data overlap with the evaluation benchmarks is not documented as disjoint, and (ii) the unqualified SOTA label is contradicted by the paper's own Table 2, where voyage-code-3 has a higher Overall AVG and MTEB Code AVG than both JCE models. These issues are load-bearing because the conclusions rest entirely on the benchmark scores.

major comments (3)
  1. [§4.1, Appendix A, Tables 3 and 4(b)] The training data description states that training includes 'the training splits of MTEB code tasks' and CoSQA+, and Table 3 lists many datasets that are also used as evaluation benchmarks in Table 2: AppsRetrieval, CodeFeedbackMT, CodeFeedbackST, CodeTransOceanContest, CodeTransOceanDL, CodeSearchNetCCRetrieval, COIR-CodeSearchNet, StackOverflowQA, SyntheticText2SQL, CoSQA, CoSQA+, and others. The paper never states that evaluation test splits were held out from training, nor that CoSQA+ has disjoint train/test splits. This is not a minor omission: if the same query–document pairs appear in training and evaluation, the reported scores, e.g. CoSQA+ 15.42%/16.38% or the MTEB averages, measure memorization rather than retrieval generalization. The authors must either document split disjointness explicitly (e.g., dataset split IDs, decontamination procedure) or rerun the evaluation on held-
  2. [Abstract, Conclusion, Table 2] The abstract and conclusion claim 'state-of-the-art performance' for the jina-code-embeddings suite. Table 2 does not support this: voyage-code-3 achieves Overall AVG 79.23% vs. JCE-1.5B 79.04%, and MTEB Code AVG 79.84% vs. JCE-1.5B 78.94%. Even setting aside the leakage concern, the unqualified SOTA claim is contradicted by the paper's own results. The claim should be revised to 'competitive with much larger models' or explicitly scoped to 'state-of-the-art among open-source models of comparable size,' with the exact ranking stated.
  3. [§5, Table 2, Appendix C] The evaluation table does not state the metric used (presumably nDCG@10 for MTEB tasks, but this should be explicit) nor which specific splits of each dataset were used. For the two large benchmarks evaluated via public MTEB scores, the authors do not state whether those public scores were computed with the same query/document prefixes and pooling method. These details matter for reproducibility and for interpreting the comparison, especially since the paper's contribution is precisely the training recipe and its interaction with evaluation.
minor comments (5)
  1. [Eq. (1)] The InfoNCE loss notation has index ranges 'i,j=0' to 'n' while the batch is defined as having n pairs; this is off by one and the rendering 'nX' is garbled. The denominator sum should be over the batch size, and the indices should be 1..n or 0..n-1 consistently.
  2. [Table 2] The table would be clearer with a row/column indicating which models are open-weight versus API-based, and which numbers are from the authors' own runs versus public leaderboard scores. The footnote lists benchmark expansions but not the evaluation metric or the exact split used for each row.
  3. [Appendix B, Table 5] The pooling ablation shows very small differences between last-token (78.41 Overall AVG) and latent attention (78.27), and some individual tasks favor latent attention. The statement in §3 that last-token pooling 'gave us better performance' is too strong without error bars or significance testing; a softer wording such as 'slightly better average performance in our experiments' is more accurate.
  4. [Table 3] CoSQA and CoSQA+ are listed as separate training sources, but the relationship between them is not explained. Since CoSQA+ is also an evaluation benchmark, the reader needs to know whether CoSQA+ is a superset of CoSQA and how the train/test splits are defined.
  5. [General] Minor typos and formatting issues: 'CosQA+' in §5 should be 'CoSQA+'; Table 2's 'CTODL*' appears in the footnote as 'CodeTransOceanDL' but not expanded in the table; the Code2Completion prefix in Table 1 has an awkward line break that obscures the intended instruction string.

Circularity Check

1 steps flagged · score 6.0 of 10

CoSQA+ is used both as a training source and as an evaluation benchmark, so the reported CoSQA+ score and the overall average are partly trained-in rather than predicted.

  1. fitted input called prediction [Section 4.1 / Appendix A / Section 5 / Table 3 / Table 2]
    "These pairs have been collected from various sources, including the training splits of MTEB code tasks and the non-MTEB code retrieval dataset CoSQA+. [...] We also evaluate the model on code-related MTEB tasks CodeSearchNetRetrieval, CodeEditSearchRetrieval, HumanEval, MBPP, DS-1000, WikiSQL, and MLQuestions, as well as CosQA+ and our in-house benchmarks."

    Section 4.1 and Appendix A identify CoSQA+ as a training source, while Section 5 and Table 2 report CoSQA+ as an evaluation benchmark. The paper never states that the evaluation uses a disjoint test split of CoSQA+; the same dataset name appears in the training table (Table 3) and the evaluation table (Table 2). The InfoNCE training objective directly optimizes the embedding similarity for the training pairs, so the CoSQA+ row (15.42%/16.38%) measures retention of the training distribution, not generalization. Since Table 2's 'Overall AVG' includes this row, the headline 'state-of-the-art performance' is partly a fitted input renamed as a prediction, rather than an independent validation.

full rationale

The core construction of the model is not circular: starting from Qwen2.5-Coder backbones, applying InfoNCE contrastive training, and evaluating on standard MTEB/CoIR tasks with explicitly named training splits is a legitimate, externally grounded pipeline. The pooling ablation in Appendix B is model selection, not circularity. The references to prior Jina work are not load-bearing for the main derivation chain. The one serious circular step is the CoSQA+ overlap: the same dataset is listed as a training source and as an evaluation benchmark, with no documented split separation. This directly inflates one row of Table 2 and, through the Overall AVG, the paper's SOTA claim. Because the central empirical claim depends in part on this trained-on evaluation set, the paper merits a partial-circularity score of 6. All other benchmark comparisons are treated as valid held-out evaluations on the evidence presented.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical benchmark claim, not a derivation, so the ledger captures the hand-set training hyperparameters and the unstated data hygiene assumptions that the reported scores depend on.

free parameters (1)
  • InfoNCE temperature τ = 0.05
    Fixed hyperparameter for contrastive loss in Eq (1); chosen by hand, no tuning details reported.
assumptions (4)
  • domain assumption The Qwen2.5-Coder backbone provides sufficient code and language knowledge for contrastive fine-tuning to yield useful embeddings.
    The paper initializes from Qwen2.5-Coder and does not ablate backbone choice (Section 3, 4).
  • domain assumption MTEB-CoIR and related benchmarks are a fair and representative measurement of code retrieval performance.
    Evaluation in Section 5 uses these benchmarks without discussing metric definition or task representativeness.
  • domain assumption Training and evaluation splits of the datasets are disjoint.
    The paper uses training splits of MTEB code tasks and CoSQA+ (Section 4.1) and evaluates on the same named benchmarks (Table 2), but never states split disjointness.
  • domain assumption GPT-4o generated synthetic training data is semantically correct.
    Synthetic datasets validated only by manual inspection of samples (Section 4.1, Appendix A).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Code Embeddings from Code Generation Models." pith.science (2026). https://pith.science/paper/Q2QABSZP

@misc{pith2026250821290,
  author       = {Pith},
  title        = {Pith review of: Efficient Code Embeddings from Code Generation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q2QABSZP}},
  note         = {Machine review of arXiv:2508.21290}
}
read the original abstract

jina-code-embeddings is a novel code embedding model suite designed to retrieve code from natural language queries, perform technical question-answering, and identify semantically similar code snippets across programming languages. It makes innovative use of an autoregressive backbone pre-trained on both text and code, generating embeddings via last-token pooling. We outline the training recipe and demonstrate state-of-the-art performance despite the relatively small size of the models, validating this approach to code embedding model construction.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. NeSy-RAG: Neuro-Symbolic RAG for Explainable Question Answering

    cs.CL 2026-08 conditional novelty 6.0 of 10

    A retrieval-augmented QA framework that synthesizes attributable Prolog modules from text chunks, retrieves Boolean predicates to build queries, and proactively asks users for missing facts, reported to beat a pure LL...

  2. CodeNib: A Multi-View Data System for Serving Repository Context to Coding Agents

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A per-commit multi-view repository-index system with incremental updates that match independent rebuilds at 8.7x/25.4x median speedups, and context policies that cut agent trajectory tokens 50-87% while holding locali...

  3. Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents

    cs.IR 2026-07 conditional novelty 6.0 of 10

    A 427-sample workflow-grounded benchmark finds complementary strengths across lexical, structural, and embedding retrieval for coding-agent context acquisition, with large residual miss rates and a selective-abstentio...

Reference graph

Works this paper leans on

16 extracted references · 2 canonical work pages · cited by 3 Pith papers

  1. [5]

    Xiaodong Gu, Hongyu Zhang, and Sunghun Kim

    URL https://arxiv.org/abs/2503.07891. Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. Deep Code Search. In Proceedings of the 40th International Conference on Software Engineering , ICSE ’18, page 933–944,

    arXiv

  2. [8]

    Edward J

    URL https://arxiv.org/abs/2506.18902. Edward J. Hu, Yelong Shen, Phillip Wallis, et al. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations, ICLR ,

    arXiv

  3. [10]

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt

    URL https://arxiv.org/abs/2409.12186. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Code- SearchNet Challenge: Evaluating the State of Semantic Code Search,

    arXiv

  4. [11]

    Aditya Kusupati, Gantavya Bhatt, Aniket Rege, et al

    URL http: //arxiv.org/abs/1909.09436. Aditya Kusupati, Gantavya Bhatt, Aniket Rege, et al. Matryoshka Representation Learning. In Proceedings of the 36th International Conference on Neural Information Processing Systems , NIPS ’22,

    arXiv 1909

  5. [13]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, et al

    URL https://arxiv.org/abs/2405.17428. Patrick Lewis, Ethan Perez, Aleksandra Piktus, et al. Retrieval-Augmented Generation for Knowledge- Intensive NLP Tasks. In Proceedings of the 34th International Conference on Neural Informa- tion Processing Systems , NIPS ’20,

    arXiv

  6. [14]

    Xiangyang Li, Kuicai Dong, Yi Quan Lee, et al

    URL https://dl.acm.org/doi/abs/10.5555/ 3495724.3496517. Xiangyang Li, Kuicai Dong, Yi Quan Lee, et al. CoIR: A Comprehensive Benchmark for Code Information Retrieval Models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers) , pages 22074–22091, 2025a. ISBN 979-8-89176-251-0. doi: 10.18653/...

    arXiv 2025

  7. [15]

    org/abs/2410.21276

    URL https://arxiv. org/abs/2410.21276. Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT- Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pages 3982–3992,

    arXiv 2019

  8. [17]

    Hongjin Su, Weijia Shi, Jungo Kasai, et al

    URL https://arxiv.org/abs/2409.10173. Hongjin Su, Weijia Shi, Jungo Kasai, et al. One Embedder, Any Task: Instruction-Finetuned Text Embeddings. In Findings of the Association for Computational Linguistics: ACL 2023 ,

    arXiv 2023

Show all 16 references
  1. [19]

    Liang Wang, Nan Yang, Xiaolong Huang, et al

    URL https://arxiv.org/abs/ 1807.03748. Liang Wang, Nan Yang, Xiaolong Huang, et al. Improving Text Embeddings with Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 11897–11916,

    arXiv

  2. [20]

    doi: 10.18653/v1/2024.acl-long

    2024 doi

  3. [23]

    6 A Training Datasets Training data for jina-code-embeddings draws on a variety of sources, described in Section 4.1

    URL https://arxiv.org/abs/2506.05176. 6 A Training Datasets Training data for jina-code-embeddings draws on a variety of sources, described in Section 4.1. • Training data splits for MTEB code tasks, and the CoSQA+ dataset. • Other public datasets adapted to our training needs...

    2024 arXiv

  4. [642]

    Wojciech Zaremba and Greg Brockman

    URL https://aclanthology.org/2024.acl-long.642/. Wojciech Zaremba and Greg Brockman. OpenAI Codex. https://openai.com/index/ openai-codex/,

    2024

  5. [2016]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova

    URL https://github.com/ ethancaballero/description2code. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of t...

    2019

  6. [2018]

    URL https://dl.acm.org/doi/10.1145/3180155.3180167

    doi: 10.1145/3180155.3180167. URL https://dl.acm.org/doi/10.1145/3180155.3180167. Michael Günther, Jackmin Ong, Isabelle Mohr, et al. Jina Embeddings 2: 8192-Token General- Purpose Text Embeddings for Long Documents. arXiv preprint arXiv:2310.19923,

  7. [2024]

    Michael Günther, Saba Sturua, Mohammad Kalim Akram, et al

    URL https://arxiv.org/abs/2310.19923. Michael Günther, Saba Sturua, Mohammad Kalim Akram, et al. jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval. arXiv preprint arXiv:2506.18902,

    arXiv

  8. [2025]

    Zhangyin Feng, Daya Guo, Duyu Tang, et al

    URL https://arxiv.org/abs/2502.13595. Zhangyin Feng, Daya Guo, Duyu Tang, et al. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536–1547,

    2020

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.