Pith. sign in

REVIEW 3 major objections 4 minor 37 references

Token Reduction Is Not Cost Reduction

T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that for API-based coding agents, context compression measured in tokens removed is not a reliable proxy for billed cost, and demonstrates with a paired, hash-frozen campaign of 2,848 provider-billed runs that an arm remov

desk verdict The most careful empirical case yet that token reduction doesn't equal cost reduction for API agents; the cache-carryover confound bounds but doesn't break the central claim. read the letter →

arxiv 2607.12161 v3 pith:M6UIQJOL submitted 2026-07-13 cs.CL

Sarel Weinberger , Amir Hozez This is my paper
classification cs.CL
keywords tokenreductionbilledcostpromptcachingcodingagentscontextcompressionpersuccessfulexecutionpairedexperimentsLLMefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that, for API-based coding agents, how many tokens a context-reduction layer removes tells you little about how much money it saves. Over a pre-specified, paired campaign of more than 2,800 actually-billed runs across 103 tasks and three models, the authors show that prompt-cache creation and reads make up roughly 87% of reconstructed cost (about 80% of the actual bill), leaving only about 6% of input cost on surfaces a user-side compression layer can touch. An arm that removed an estimated 38% of raw tool-output tokens ended up with 6.8% higher paired billed cost, because its trajectory changed and the retrieval-stage savings were repaid by extra diagnosis and testing turns. A separate single-shot study showed compression can also destroy the byte-exact anchors an editing agent needs to apply patches, cutting successful patch application from 27/40 to 15/40. The paper concludes that context-reduction systems should be judged by success-adjusted billed cost, not token reduction.

What carries the argument

The load-bearing object is a four-component cost decomposition: uncached input, cache-creation, cache-read, and generated-output tokens, each priced with the provider's published multipliers (cache reads at 0.1×, cache writes at 1.25× the input price), calibrated so that per-run reconstruction residuals are near zero. On top of this sits the 'addressable share'—the fraction of input cost a user-side layer can modify (≈6%)—and the success-adjusted cost per successful execution, which keeps failed runs in the cost numerator. The paired block design (all arms from identical fresh working copies, randomized order) and the hook-side ledger that makes raw versus delivered tool-output tokens observ

What would settle it

Run the same paired campaign with an enforced delay greater than the five-minute cache TTL between runs in a block (or with provider-side caching disabled), and compare the arm-level cost deltas to the published ones: if the +6.8% penalty on the 38%-reduction arm and the near-null delta on the deterministic compressor shift toward zero or reverse, cache carryover, not compression policy, produced the headline decoupling. A cheaper check: inspect per-request cache-hit fields to see whether later arms in a block reuse prefixes written by earlier arms.

Watch

Extended reading notes

Core claim

The central claim is that token reduction and cost reduction decouple in API-based coding agents: in the evaluated workloads, cache traffic—not generated output—dominates the bill, and the portion of input cost a user-side layer can modify (tool outputs, retrieved files, conversation history) sums to roughly 6%, putting a ~5% ceiling on visible-token compression before trajectory effects are considered. Empirically, removing 38.4% of estimated raw tool-output tokens coincided with a +6.8% paired cost increase (95% CI [+2.8%, +11.3%]), and per-task reduction predicted cost change only weakly (Pearson r=0.15, CI crossing zero). Compression also degraded task success when it corrupted load-bear

Load-bearing premise

The argument collapses if within-block prompt-cache carryover between arms systematically favored or penalized particular arms: because every consecutive run pair falls inside the provider's five-minute cache TTL, and per-run home-directory isolation does not isolate the provider's cache, the reported arm cost differences could reflect shared cached prefixes rather than the interventions themselves.

Editorial extensions

If this is right

  • Component-level token-reduction gains should no longer be reported as efficiency; benchmarks should report paired, actually-billed cost per successful execution with confidence intervals.
  • User-side compression layers in single-prompt benchmark harnesses have a hard ceiling around 5% of input cost, because ~94% of input cost is locked in system prompts, tool schemas, and hidden thinking.
  • Any compression that lengthens the trajectory—extra diagnosis, re-retrieval, re-reading—tends to repay per-turn savings at full cache prices, so trajectory-neutral compression is the only kind with a clear cost case.
  • Aggressive compression of dense, load-bearing evidence (tracebacks, test output, exact edit anchors) can reduce task success even when token metrics look good; content-aware preserve gates are required.
  • The same compression layer can be a cost penalty on one agent harness and a cost saving on another, so efficiency claims must be re-measured per deployment rather than assumed to transfer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's single-prompt benchmark harness leaves roughly 94% of input cost in framework-owned surfaces; the authors' own analysis of interactive sessions suggests the accessible surface may grow to ~30% of input cost, implying compression layers could matter more in interactive use than in the benchmark runs that anchor this study.
  • The measured cost anatomy is tied to one provider's July-2026 cache pricing and five-minute TTL; if cache-read discounts deepen or TTLs lengthen, the balance between cache traffic and output tokens will shift, and the 'addressable share' ceiling would move accordingly.
  • A natural testable extension is to instrument per-request cache-prefix reuse across arms (or to rerun with inter-arm delays beyond the TTL) to directly quantify carryover, which the authors could only bound with order-sensitivity checks.
  • Because per-task reduction correlated weakly with cost change, designing a 'cost-aware' compression policy that predicts which tasks reward compression—rather than maximizing tokens removed—is the direct practical next step the evidence supports.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper reports a large, pre-specified, paired, provider-billed campaign of Claude Code runs (2,908 executed; 2,848 analyzed) comparing a baseline against two hook-based compression generations (RTK, RTK-ML) and an API-boundary proxy (Headroom). The central empirical findings are: (i) prompt-cache creation and reads account for roughly 87% of the reconstructed four-component cost (about 80% of the actual bill, with an 8.7% dollar-weighted residual); (ii) local tool-output reduction does not predict end-to-end billed cost—an arm removing 38.4% of estimated raw tool-output tokens had a paired cost increase of +6.8% (95% CI [+2.8%, +11.3%]); and (iii) aggressive compression can damage task success by corrupting verbatim edit anchors in a small single-shot study. The paper also proposes a layered evidence taxonomy (L1–L8) and argues for success-adjusted billed cost as the decision-grade metric. The manuscript is exceptionally transparent: it includes frozen manifests, append-only ledgers, retained transcripts, explicit limitations, and a post-hoc replication with the unmodified upstream RTK distribution that reproduces the null.

Significance. If the results stand, the paper provides an important correction to the common practice of evaluating context-compression layers by token-reduction ratios. It supplies one of the largest publicly described, actually-billed, paired datasets for coding-agent cost measurement, with careful randomization, provider-returned total_cost_usd, bootstrap CIs, and a reproducibility package. The explicit handling of the unresolved 8.7% residual and the cross-agent Codex replication (sign reversal for Headroom) strengthen the paper's central message that efficiency claims must be measured end-to-end. The empirical cost anatomy and the layered evidence standard are valuable contributions to the community, even if some specific point estimates carry uncertainty.

major comments (3)
  1. [§5.3, Table 6] The prompt-cache carryover threat is load-bearing for both the +6.8% RTK-ML delta and the ~87% cache-share composition, yet the checks in Table 6 are indirect and underpowered. The RTK-ML order difference (+7.49 points, CI [−2.32,+18.01]) is in the direction carryover would predict and its confidence interval is wide, so the null result does not rule out the confound. The 'no cache-write deflation at later positions' check is not a direct test: a later arm can reuse an earlier cached prefix and still generate cache-creation tokens in later turns after the prompt diverges. Because 100% of the 2,136 consecutive run pairs fall inside the 5-minute TTL and $HOME isolation does not isolate the provider-side cache, the retained per-turn ledgers should allow a direct analysis of first-request cache_read_input_tokens and cache_creation_input_tokens by block position and by arm order. Please add t
  2. [§6.1, Eq. (1), Table 7] The 8.7% dollar-weighted residual is not a small accounting artifact: on Haiku 4.5 it scales monotonically with the thinking-effort setting, while measured output and cache volumes stay flat. This implies a systematic omitted cost component in the four-component reconstruction. Consequently, the '87% of reconstructed cost' figure is a share of a denominator that excludes this residual, and the normalized four-component shares in Appendix A.3 (Table 21) remove it entirely. This is not merely a presentation issue because the paper's headline—'cache creation and reads accounted for approximately 87%'—could be overstated relative to the actual bill (about 80%, as the paper notes). Please present all composition tables and figures on a consistent billed-cost denominator that includes the residual, or clearly label the denominator in every instance, including the normalized four-component shar
  3. [§7.2, Fig. 4] The claim that 'component-level reduction did not predict end-to-end cost movement' rests on a per-task correlation (r = 0.154, CI [−0.051, +0.356]) computed on observed tool-output reduction, which is endogenous to the arm's own trajectory: the RTK-ML arm can change commands, search behavior, and turns, so the 'reduction' is not an assigned dose. The paper acknowledges this, but the RQ3 answer in the abstract and conclusion is stated more strongly than the evidence supports. Additionally, 87 of 100 tasks had <0.1% reduction, so the correlation is dominated by a narrow range. Please add a sensitivity analysis restricted to tasks with meaningful reduction (e.g., the 5–20% band, currently n=4), report binned means with CIs, and soften the causal-sounding phrasing in the abstract/conclusion.
minor comments (4)
  1. [Table 6] Please specify the sign convention for 'Order difference' in the caption (arm-first minus arm-after, or vice versa). The current text states it as 'arm before vs. after' but the reader must reconstruct the direction from the numbers.
  2. [§5.3] The phrase 'each block position contains exactly 712 runs' is confusing because a block has four positions and 712 blocks, so each position has 712 runs by construction. The useful fact is the lack of position-stratified arm counts; consider clarifying that arm counts at each position were not stratified but vary 147–221 under randomization.
  3. [§7.4] Typo/formatting: 'failure mixedit_apply_failed 20 vs. 12' should read 'failure mix: edit_apply_failed 20 vs. 12' or similar.
  4. [§2.3] The 'working ceiling for visible-token compression is ≈5% of input cost' is presented as a general bound, but it is derived from the benchmark corpus; the interactive-session composition in §14 indicates a much larger accessible surface (~30%). Please explicitly label this as a benchmark-corpus bound in the main text, not a universal ceiling.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are direct empirical measurements from provider-billed runs, with published cache pricing and a post-hoc upstream replication.

full rationale

The paper's load-bearing claims are empirical, not derived. The cost reconstruction in eq. (1) uses published Anthropic multipliers (µ_w=1.25, µ_r=0.1), and section 6.1 calibrates it against actual billed amounts while reporting median and aggregate residuals; therefore the ~87% cache-share figure is a measurement with disclosed error, not an output forced by the equation. The headline decoupling result—RTK-ML removing 38.4% of raw tool-output tokens while paired billed cost rose +6.8% (95% CI [+2.8,+11.3])—is a paired bootstrap estimate from table 8, not a consequence of any fitted parameter; the token reduction comes from the hook-side ledger and the cost comes from provider total_cost_usd. The table-1 'accessible ceiling' is a definitional upper bound: by construction, a layer can save at most the cost of the surfaces it can touch. The paper labels this as a frame, not as a prediction. The only author-involvement concern is that the RTK arms were built by the authors, but section C adds a checksum-verified upstream RTK v0.44.1 replication that reproduces the null on previously unmeasured Opus cells, so the central result does not reduce to the authors' own construction. The prompt-cache carryover discussion in section 5.3 is an explicit validity limitation, not a circularity: it questions whether the measured arm differences are unbiased, but does not make any claim identical to its inputs. No uniqueness theorem is imported from the authors' prior work, no fitted parameter is renamed as a prediction, and no known result is merely relabeled.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on four domain assumptions: usage-field faithfulness, absence of cache-carryover bias, deterministic judges as success, and local tokenizer comparability. No free parameters are fitted to the target result; provider cache multipliers are external inputs. No new physical or conceptual entities are postulated.

assumptions (4)
  • domain assumption Provider usage fields (input_tokens, cache_creation_input_tokens, cache_read_input_tokens, output_tokens) and published list prices with mu_r=0.1, mu_w=1.25 fully determine billed cost per run; any residual is separately attributable.
    Section 6.1, eq. (1); calibration leaves an 8.7% dollar-weighted residual and per-model effort-scaling on Haiku, so the assumption is partially violated and flagged by the authors.
  • domain assumption Cross-arm prompt-cache carryover within a block does not systematically bias paired arm deltas.
    Section 5.3: all within-block gaps are under the 5-minute cache TTL; randomization and null order-sensitivity checks bound but cannot exclude carryover.
  • domain assumption Deterministic script judges and pytest/diff outcomes are a valid measure of task success for efficiency comparisons.
    Sections 5.2 and 14; quality dimensions such as maintainability, patch quality, and reasoning quality are not measured.
  • domain assumption The local tiktoken o200k_base BPE tokenizer estimates raw-vs-delivered tool-output tokens comparably to the provider tokenizer for the 38.4% reduction claim.
    Section 7.2; the authors label these as estimates, not provider token counts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Token Reduction Is Not Cost Reduction." pith.science (2026). https://pith.science/paper/M6UIQJOL

@misc{pith2026260712161,
  author       = {Pith},
  title        = {Pith review of: Token Reduction Is Not Cost Reduction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M6UIQJOL}},
  note         = {Machine review of arXiv:2607.12161}
}
read the original abstract

Context-reduction layers for API-based coding agents, including command-output compressors, retrieval rankers, and API-boundary proxies, are commonly evaluated by how much context or tool output they remove. We ask a different question: which interventions actually reduce end-to-end billed cost while preserving task success? Our primary evidence is a pre-specified, hash-frozen, paired campaign of 2,908 provider-billed Claude Code runs, of which 2,848 were analyzed, covering 103 tasks, seven repositories, and three models. The campaign compared a baseline with two generations of hook-based compression and an API-boundary proxy within a broader measured program of roughly 5,500 billed executions. Three findings emerge. First, prompt-cache traffic dominated cost composition, accounting for about 87% of reconstructed four-component cost (about 80% of the actual bill), with an 8.7% dollar-weighted residual not attributable from retained telemetry. Second, local payload reduction was not a reliable predictor of end-to-end billed cost. An arm that removed 38% of estimated raw tool-output tokens incurred 6.8% higher paired cost (95% CI: +2.8% to +11.3%), while per-task reduction showed only a weak association with cost change (Pearson r = 0.15). Third, aggressive compression can remove action-critical evidence: on SWE-bench-derived Go tasks, compression reduced successful patch application from 27/40 to 15/40 by corrupting verbatim edit anchors. We propose evaluating context-reduction systems by success-adjusted billed cost rather than token reduction alone.

Figures

Figures reproduced from arXiv: 2607.12161 by the authors.

Figure 1
Figure 1. Cost decomposition as shares of actual billed cost (n=2,848 analyzed runs; component prices as in eq. (1)). The hatched fifth segment is the unattributed billed-cost residual (billed minus reconstructed). Numbers inside segments are percentages; run-bootstrap 95% CIs for every segment are in table 6. Normalized four-component shares appear in Appendix A.3 [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Per-task estimated raw tool-output reduction (local BPE tokenizer) vs. paired billed-cost change (RTK-ML arm, Haiku 4.5; each point one task, n = 100; task means over paired blocks; red squares: binned means with bootstrap 95% CIs). Pearson r = 0.154 [-0.051, +0.356]; Spearman ρ = 0.013 [-0.082, +0.339] (task bootstrap, 10,000 resamples, seed 7). Observed reduction is endogenous to the arm’s own trajectory, not an a… view at source ↗
Figure 2
Figure 2. Realized reduction per accessible component. Light bars: each component’s share of input cost. Overlaid bars: the share actually removed by the evaluated hook layers, applying the hook-side ledger’s raw-vs-delivered tool-output reduction (−38.4% RTK-ML, −1.3% RTK; section 7.2) to the tool-output share. Even the campaign’s largest observed reduction reaches ≈1.3% of input cost—and that arm’s paired billed cost was hi… view at source ↗
Figures from the paper (15 more)
Figure 3
Figure 3. Figure 3: Cost per successful execution by system and model (whiskers: task-bootstrap 95% CIs; 10,000 resamples, seed 7; success and run counts in table 9) [PITH_FULL_IMAGE:figures/full_fig_p014_3.png]
Figure 3
Figure 3. Figure 3: Cost decomposition as shares of actual billed cost (n=2,848 analyzed runs; component prices as in eq. (1)). The hatched fifth segment is the unattributed billed-cost residual (billed minus reconstructed). Numbers inside segments are percentages; run-bootstrap 95% CIs f…
Figure 4
Figure 4. Figure 4: Billed cost vs. trajectory length (assistant turns) per run, by model (n=2,848 runs; log-scaled cost). Each added turn re-transmits the cached prefix, so trajectory changes dominate per-turn savings [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 4
Figure 4. Figure 4: Per-task estimated raw tool-output reduction (local BPE tokenizer) vs. paired billed-cost change (RTK-ML arm, Haiku 4.5; each point one task, n = 100; task means over paired blocks; red squares: binned means with bootstrap 95% CIs). Pearson r = 0.154 [-0.051, +0.356]; …
Figure 5
Figure 5. Figure 5: Paired billed-cost change vs. baseline by model–effort cell (task-clustered 95% CIs). Unmeasured cells (Sonnet 5 / Opus 4.8 at medium, xhigh, max) are annotated, not imputed. with 131/132 task success. The qualitative pattern—no free lunch from compression on long sess…
Figure 6
Figure 6. Figure 6: Paired billed-cost change by task family (task counts in parentheses; CIs where ≥3 tasks). Headroom is right of zero in every family; RTK effects change sign across families [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 6
Figure 6. Figure 6: Billed cost vs. trajectory length (assistant turns) per run, by model (n=2,848 runs; log-scaled cost). Each added turn re-transmits the cached prefix, so trajectory changes dominate per-turn savings [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Generated-token composition by phase (characterization split; 10,376 turns). Coarse phase-token shares were similar across arms under the frozen turn-level taxonomy. than in the measured cost and context volume associated with those phases (delivered tool bytes and the…
Figure 7
Figure 7. Figure 7: Paired billed-cost change vs. baseline by model–effort cell (task-clustered 95% CIs). Unmeasured cells (Sonnet 5 / Opus 4.8 at medium, xhigh, max) are annotated, not imputed. 9 Cross-Agent Replication on Codex A subsequent replication campaign (codex_headroom_replicati…
Figure 8
Figure 8. Figure 8: Left: median cache-read context at the first code edit (runs with an implementation phase, base splits, all models). Right: median estimated delivered tool tokens before the first edit (Haiku runs; bytes/4 estimate). Headroom arrives at the first edit ≈49% heavier; the…
Figure 8
Figure 8. Figure 8: Paired billed-cost change by task family (task counts in parentheses; CIs where ≥3 tasks). Headroom is right of zero in every family; RTK effects change sign across families. Result. Both arms passed 39/40 tasks with no discordant correctness outcome (both failed the s…
Figure 9
Figure 9. Figure 9: Layered evidence taxonomy for token-efficiency claims. Every system in table 1 is placed at the highest layer its retained artifacts support. Embedding retrieval. The BGE embedding leg was enabled, health-checked, and outage-free throughout the authoritative campaign, …
Figure 9
Figure 9. Figure 9: Generated-token composition by phase (characterization split; 10,376 turns). Coarse phase-token shares were similar across arms under the frozen turn-level taxonomy [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Left: median cache-read context at the first code edit (runs with an implementation phase, base splits, all models). Right: median estimated delivered tool tokens before the first edit (Haiku runs; bytes/4 estimate). Headroom arrives at the first edit ≈49% heavier; th…
Figure 11
Figure 11. Figure 11: Layered evidence taxonomy for token-efficiency claims. Every system in table 2 is placed at the highest layer its retained artifacts support. small single-shot study, and a plausible-looking ratio metric (successes per million tokens) would have inverted the verdict; …

Review history (3 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 15 linked inside Pith

  1. [1]

    LoCoEval: Long-conversation evaluation pipeline.https://anonymous.4open.science/ r/LoCoEval, 2026

    Anonymous. LoCoEval: Long-conversation evaluation pipeline.https://anonymous.4open.science/ r/LoCoEval, 2026. Anonymized research artifact; accessed 2026-06

    2026

  2. [2]

    Prompt caching — claude developer platform documentation.https://platform.claude

    Anthropic. Prompt caching — claude developer platform documentation.https://platform.claude. com/docs/en/build-with-claude/prompt-caching, 2026. Accessed 2026-07. 30

    2026

  3. [3]

    Claude code documentation.https://code.claude.com/docs, 2026

    Anthropic. Claude code documentation.https://code.claude.com/docs, 2026. Accessed 2026-07

    2026

  4. [4]

    Claude developer platform pricing

    Anthropic. Claude developer platform pricing. https://platform.claude.com/docs/en/pricing,

  5. [5]

    ast-grep: A CLI tool for code structural search, lint, and rewriting

    ast-grep contributors. ast-grep: A CLI tool for code structural search, lint, and rewriting. https: //ast-grep.github.io, 2026. Version 0.43.0; accessed 2026-07

    2026

  6. [6]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  7. [7]

    LongBench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. arXiv:2308.14508

    arXiv 2024

  8. [8]

    FrugalGPT: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176, 2023

    Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176, 2023

    arXiv 2023

Show all 37 references
  1. [9]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  2. [10]

    ContextBench dataset

    ContextBench contributors. ContextBench dataset. Hugging Face datasetContextbench/ContextBench, configuration contextbench_verified, 2026. Dataset revision hash not recorded in the retained artifacts; row identifiers listed in the reproducibility appendix

    2026

  3. [11]

    SWE-bench_pro dataset (go split)

    ContextBench contributors. SWE-bench_pro dataset (go split). Hugging Face dataset Contextbench/SWE-bench_Pro, test split, 2026. Accessed 2026-05; Go rows selected by repository language

    2026

  4. [12]

    CodeBERT: A pre-trained model for programming and natural languages

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A pre-trained model for programming and natural languages. InFindings of the Association for Computational Linguistics: EMNLP 2020, 2020

    2020

  5. [13]

    ripgrep: Recursively search directories for a regex pattern.https://github.com/ BurntSushi/ripgrep, 2026

    Andrew Gallant. ripgrep: Recursively search directories for a regex pattern.https://github.com/ BurntSushi/ripgrep, 2026. Accessed 2026-07

    2026

  6. [14]

    Prompt cache: Modular attention reuse for low-latency inference

    In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. InProceedings of Machine Learning and Systems (MLSys), 2024

    2024

  7. [15]

    Headroom: An API-boundary context optimization proxy

    Headroom Labs. Headroom: An API-boundary context optimization proxy. https://github.com/ headroomlabs-ai/headroom, 2026. Version 0.27.0; open-source distribution; accessed 2026-07

    2026

  8. [16]

    Code- SearchNet challenge: Evaluating the state of semantic code search.arXiv preprint arXiv:1909.09436, 2019

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Code- SearchNet challenge: Evaluating the state of semantic code search.arXiv preprint arXiv:1909.09436, 2019

    1909 arXiv

  9. [17]

    LLMLingua: Compressing prompts for accelerated inference of large language models

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LLMLingua: Compressing prompts for accelerated inference of large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  10. [18]

    LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression

    Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguis...

    2024

  11. [19]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? InInternational Conference on Learning Representations (ICLR), 2024. arXiv:2310.06770. 31

    2024 arXiv

  12. [20]

    Efficient memory management for large language model serving with PagedAttention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles (SO...

    2023

  13. [21]

    Retrieval-augmented generation for knowledge-intensive NLP tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. InAdvances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  14. [22]

    Compressing context to enhance inference efficiency of large language models

    Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  15. [23]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024

    2024

  16. [24]

    Learning to compress prompts with gist tokens

    Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  17. [25]

    Introducing SWE-bench Verified

    OpenAI. Introducing SWE-bench Verified. https://openai.com/index/ introducing-swe-bench-verified/, 2024. Accessed 2026-07

    2024

  18. [26]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  19. [27]

    Efficient large language models: A survey

    Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, and Mi Zhang. Efficient large language models: A survey. Transactions on Machine Learning Research, 2024. arXiv:2312.03863

    2024 arXiv

  20. [28]

    Executable code actions elicit better LLM agents

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better LLM agents. InInternational Conference on Machine Learning (ICML), 2024. arXiv:2402.01030

    2024 arXiv

  21. [29]

    Xu, Xiangru Tang, Mingchen Zhuge, et al

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, et al. OpenHands: An open platform for AI software developers as generalist agents.arXiv preprint arXiv:2407.16741, 2024

    2024 arXiv

  22. [30]

    C-Pack: Packaged resources to advance general Chinese embedding.arXiv preprint arXiv:2309.07597, 2023

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-Pack: Packaged resources to advance general Chinese embedding.arXiv preprint arXiv:2309.07597, 2023. BGE embedding model family

    2023 arXiv

  23. [31]

    Retrieval meets long context large lan- guage models

    Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large lan- guage models. InInternational Conference on Learning Representations (ICLR), 2024. arXi...

    2024 arXiv

  24. [32]

    InterCode: Standardizing and benchmarking interactive coding with execution feedback

    John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. InterCode: Standardizing and benchmarking interactive coding with execution feedback. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2306.14898

    2023 arXiv

  25. [33]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2405.15793

    2024 arXiv

  26. [34]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  27. [35]

    τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024. 32

    2024 arXiv

  28. [36]

    Multi-SWE-bench: A multilingual benchmark for issue resolving.arXiv preprint arXiv:2504.02605, 2025

    Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, et al. Multi-SWE-bench: A multilingual benchmark for issue resolving.arXiv preprint arXiv:2504.02605, 2025

    2025 arXiv

  29. [37]

    18/21” and “21/21

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. RepoCoder: Repository-level code completion through iterative retrieval and generation. InProceedings of the 2023 Conference on Empirical Methods in Natural Language...

    2023

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.