REVIEW 3 major objections 4 minor 2 cited by
The Hidden Cost of Readability: How Code Formatting Silently Consumes Your LLM Budget
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Removing code formatting cuts LLM input tokens by 24.5% without hurting code completion performance, this paper argues.
desk verdict The paper quantifies a practical cost-saving (24.5% input tokens) for removing formatting in FIM completion across 10 models, but the Python indentation question and lack of error bars in the abstract need checking before the safety claim is taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is the linear tokenization of source code: indentation and newlines become discrete tokens that consume the LLM's context window and compute budget but, the paper argues, carry little task-relevant signal for completion. The load-bearing device is a bidirectional code transformation tool that removes formatting before inference and can restore it afterward, plus prompting and fine-tuning strategies that teach models to emit compact code without sacrificing correctness.
What would settle it
Run a head-to-head completion benchmark on the same four languages with execution-based tests and semantic-equivalence checks: if any of the ten models shows a statistically significant drop in pass@k or in compile-and-test success when formatting is stripped, the paper's 'maintain performance' claim would fail for that model. A second falsifier: repeat the prompting and fine-tuning experiment but measure human-rated readability of the shorter outputs; if the 36.1% shorter code is consistently less maintainable, the 'without compromising correctness' claim is incomplete.
Extended reading notes
Core claim
The central claim is that formatting elements behave like visual aids for humans but are essentially inert tokens for LLMs in code completion. On Fill-in-the-Middle tasks, replacing formatted code with a compact unformatted version keeps model output quality largely unchanged, while trimming the input token stream by about a quarter. A further result is that when models are prompted or fine-tuned to write unformatted code directly, their generated responses can be up to 36.1% shorter, again without a measured correctness penalty. The paper positions this as a bidirectional code transformation opportunity: one tool can strip formatting for the model and restore it for human viewing, so readab
Load-bearing premise
The study assumes that completion-task parity across four languages generalizes to all code workloads, and that its correctness metrics would catch every meaningful loss from removing formatting.
Editorial extensions
If this is right
- If the result holds, any LLM-based code completion service can strip formatting from repository context before inference and cut roughly a quarter of input token cost with no expected drop in completion accuracy.
- The reported up-to-36.1% output reduction from prompting or fine-tuning translates directly into lower latency and lower per-request cost for generated code.
- Formatting removal is orthogonal to model choice, since the parity was observed across ten commercial and open-source models, so providers can adopt it without retraining.
- The bidirectional transformation approach removes the usability objection: developers can continue to read formatted code while the LLM operates on compact code.
Reading between the lines
- The paper leaves open whether the same token savings transfer to code tasks beyond completion, such as bug localization, code review, or repository-level reasoning; a natural extension is to test format-stripped prompts across those tasks.
- If the parity is robust, the 24.5% input saving compounds with context-caching systems because shortened prompts also reduce cache size and retrieval cost, not just one-shot inference.
- The reported output shortening from fine-tuning suggests a training objective that penalizes formatting tokens could yield even larger savings than prompt-based stripping, since the model learns to avoid emitting them altogether.
- A cautious reader should treat the correctness guarantee as limited to the metrics used in the study; evaluating with human-judged code quality or execution-based tests would test whether subtle degradation hides behind aggregate scores.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims that code formatting elements (indentation, newlines) are largely non-essential for LLM code processing, and that removing them before inference reduces input tokens by an average of 24.5% while preserving performance on Fill-in-the-Middle code completion across Java, Python, C++, and C# with ten models. It further reports that prompting and fine-tuning can reduce output code length by up to 36.1% without correctness loss, and introduces a bidirectional transformation tool to integrate format removal into inference workflows.
Significance. If the central claim holds, the paper identifies a simple, high-impact efficiency optimization for LLM-based code completion: strip human-oriented formatting before inference and restore it afterward. The study's breadth (four languages, ten models) and the practical tool are genuine strengths. However, the claim is currently under-specified and potentially over-generalized: the evidence is limited to FIM completion, the correctness metric is unnamed, and the treatment of Python indentation—which is syntactically significant—poses a direct internal validity question. The practical significance is real but contingent on resolving these issues.
major comments (3)
- [Abstract] The key claim that removing 'indentation and newlines' from Python code maintains LLM performance is not plausible as stated. In Python, indentation is lexically significant: it produces INDENT/DEDENT tokens and defines block structure. Stripping indentation yields syntactically invalid inputs; preserving minimal indentation changes what 'unformatted' means and lowers the achievable token reduction. The abstract's average 24.5% token reduction therefore cannot be evaluated without per-language numbers and a precise description of the transformation for Python. Please report per-language token reductions and demonstrate that transformed Python inputs remain valid and semantically equivalent to the originals.
- [Abstract (scope)] The conclusion that formatting elements 'do not seem to be beneficial for LLMs' and the recommendation to remove them as a practical optimization are derived exclusively from Fill-in-the-Middle completion tasks. The abstract frames the study as 'comprehensive' without this scope qualification. If formatting carries task-relevant signal in, e.g., debugging, code review, or structural reasoning, the efficiency recommendation would overstate its safety. Please either narrow the claims to FIM completion or add evidence from additional code tasks.
- [Abstract (correctness and output savings)] The phrase 'without compromising correctness' and the later 'up to 36.1%' output reduction require a defined correctness metric (e.g., exact match, pass@k, unit-test pass rate) and statistical support. As written, the abstract gives no basis for judging whether parity is measured with sufficient sensitivity, especially for the fine-tuning experiments where benchmark overlap and test distribution could confound the result. Report the metric, confidence intervals, and significance tests for the performance-parity claim.
minor comments (4)
- [Abstract] 'Negligible output token reductions' is vague; provide the actual figure or range in the abstract.
- [Abstract] The list of 'ten LLMs' is not given; even a parenthetical enumeration would help the reader judge model diversity.
- [Abstract] The bidirectional transformation tool is described only at a high level; clarify whether it is a formatter-aware tokenizer wrapper, a pre/post-processing pipeline, or a fine-tuning data transformation.
- [Abstract] The 'up to 36.1%' output reduction conflates prompting and fine-tuning results; distinguish the two conditions and their respective gains.
Circularity Check
No circularity: the abstract reports direct empirical measurements against external benchmarks; no fitted parameter is relabeled as a prediction, and no load-bearing self-citation appears.
full rationale
This is an abstract-only review of an empirical benchmark study. The central claims—that LLM performance is maintained when formatting is removed, that token counts drop by 24.5% on average, and that prompting/fine-tuning can reduce output length up to 36.1%—are presented as measured experimental outcomes, not as derivations from a model whose parameters are fit to the same data. There is no equation in the abstract that reduces to an identity, no parameter fitted to a subset and then called a prediction, and no invocation of a uniqueness theorem or prior self-citation to force a conclusion. The skeptical concern about Python indentation being syntactic rather than purely formatting is a potential internal-validity issue, not a circularity issue: it questions whether the measured parity is meaningful, but it does not show that the conclusion is equivalent to the input by construction. Even the 'up to 36.1%' figure, while it could in principle be the result of test-set tuning, is not described as such, and no evidence in the abstract supports that circular mechanism. Under the hard rule that circularity must be exhibited by quote and specific reduction, no circular step can be identified from the available text. The appropriate finding is therefore no significant circularity.
Assumptions & free parameters
assumptions (2)
- domain assumption Fill-in-the-Middle code completion tasks (with their correctness metric) are representative of the code workloads where LLM formatting effects would appear.
- domain assumption The chosen correctness metric is sensitive enough to detect any quality loss caused by removing formatting tokens.
Cite this review
Pith. "Pith review of The Hidden Cost of Readability: How Code Formatting Silently Consumes Your LLM Budget." pith.science (2026). https://pith.science/paper/SH4VL7WU
@misc{pith2026250813666,
author = {Pith},
title = {Pith review of: The Hidden Cost of Readability: How Code Formatting Silently Consumes Your LLM Budget},
year = {2026},
howpublished = {\url{https://pith.science/paper/SH4VL7WU}},
note = {Machine review of arXiv:2508.13666}
}
read the original abstract
Source code is usually formatted with elements like indentation and newlines to improve readability for human developers. However, these visual aids do not seem to be beneficial for large language models (LLMs) in the same way since the code is processed as a linear sequence of tokens. Furthermore, these additional tokens can lead to increased computational costs and longer response times for LLMs. If such formatting elements are non-essential to LLMs, we can reduce such costs by removing them from the code. To figure out the role played by formatting elements, we conduct a comprehensive empirical study to evaluate the impact of code formatting on LLM performance and efficiency. Through large-scale experiments on Fill-in-the-Middle Code Completion tasks across four programming languages (Java, Python, C++, C\#) and ten LLMs-including both commercial and open-source models-we systematically analyze token count and performance when formatting elements are removed. Key findings indicate that LLMs can maintain performance across formatted code and unformatted code, achieving an average input token reduction of 24.5\% with negligible output token reductions. This makes code format removal a practical optimization strategy for improving LLM efficiency. Further exploration reveals that both prompting and fine-tuning LLMs can lead to significant reductions (up to 36.1\%) in output code length without compromising correctness. To facilitate practical applications, we develop a bidirectional code transformation tool for format processing, which can be seamlessly integrated into existing LLM inference workflows, ensuring both human readability and LLM efficiency.
Forward citations
Cited by 2 Pith papers
-
CodeShrink: Adaptive Visual Compression for Efficient Multimodal Code Understanding
A three-part pipeline (blank-free rendering, instruction-aware visual-token pruning, and per-input compression settings) cuts visual tokens for code-image MLLM inputs by up to 71% without losing accuracy.
-
Characterizing Readability Issue Patterns and the Role of Prompt Design in LLM-Generated Code
Using a 61-feature readability model, LLM code matches or slightly exceeds human code in readability score, shows distinct issue patterns, and prompt design has limited influence.
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.