Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

An Enhanced Text Compression Approach Using Transformer-based Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that deleting all vowels from English text and compressing with LZW reaches compression ratios of 11.42 to 13.38 on three corpora, with transformers able to restore the missing vowels.

desk verdict The claimed SOTA compression ratios fail because they compare original text length against LZW-compressed vowel-stripped text while omitting the 63M-parameter restoration model, so the central result is unsupported; the restoration experiments are honest but the paper is not ready for publication as is. read the letter →

arxiv 2412.15250 v1 pith:UKN4CYQG submitted 2024-12-15 cs.CL cs.ITcs.LGmath.IT

Chowdhury Mofizur Rahman , Mahbub E Sobhani , Anika Tasnim Rodela , Swakkhar Shatabda This is my paper · ORCID
classification cs.CLcs.ITcs.LGmath.IT
keywords textcompressionvowelremovalLempel-Ziv-WelchtransformerrestorationBLEUBERTScore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that a simple preprocessing step, deleting every vowel from English text, changes the compression problem enough that a standard LZW compressor reaches compression ratios around 12 to 13 times on BookCorpus, WMT14 EN-DE, and WMT14 EN-FR. A transformer-based model called RejuvenateFormer, or the pretrained T5-Small, then reconstructs the missing vowels with moderate to high BLEU and BERTScore. If true, the practical consequence is that English text can be stored or transmitted at roughly eight percent of its original byte size on the paper's metric, at the cost of needing a neural restoration stage. The authors also claim that the preprocessing is compatible with any transformer encoder-decoder and that larger training corpora improve restoration quality.

What carries the argument

The load-bearing object is a three-stage pipeline written as Equation (1): $S_{RCI} = LZW_D(LZW_C(VR(S)))$, where $VR(\cdot)$ removes the ten English vowel characters, $LZW_C$ compresses the vowel-free string losslessly and defines the reported compression ratio, and $LZW_D$ returns the vowel-free text. A transformer decoder then generates the missing vowels autoregressively using multi-head self-attention over the reconstructed vowel-free sequence. This setup separates the lossless part, handled by LZW, from the lossy part, vowel deletion plus neural restoration, which is what makes the high compression ratios possible.

What would settle it

Take a fixed English corpus, encode it as the compressed vowel-free text plus the weights of the restoration model, divide the total bytes by the original size, and compare with a standard lossless compressor such as GZIP on the same corpus; if the end-to-end number is not smaller, the central compression-ratio claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that vowel removal is an effective lossy front-end for English text compression. The paper reports compression ratios of 12.57 on BookCorpus, 13.38 on EN-DE, and 11.42 on EN-FR, computed as the original text length divided by the LZW-compressed length of the vowel-removed text, and presents these as beating a GPT-based compression approach, TRACE, and general-purpose LZMA, GZIP, ZLIB, and arithmetic-coding baselines. The discarded vowels are then treated as a sequence-generation task: RejuvenateFormer, a six-layer encoder-decoder transformer with a 512-dimensional hidden state, is trained to map vowel-free sequences back to complete English sentences, achieving BLEU scores of 50.45 on BookCorpus, 27.31 on EN-DE, and 25.78 on EN-FR. The paper's own results also show that the pretrained T5-Small restores vowels more accurately than RejuvenateFormer on all three corpora, which the authors take as evidence that the preprocessing is model-agnostic.

Load-bearing premise

The reported compression ratio divides the original text length by the compressed vowel-free text length, without counting the size of the restoration model or the fact that the vowel-free text alone cannot be turned back into the original text; on a fair end-to-end metric this advantage would shrink.

Editorial extensions

If this is right

  • On the paper's own metric, English text can be represented at roughly eight percent of its original size by storing only the LZW-compressed vowel-free form and relying on a restoration model at use time.
  • Because T5-Small outperforms the from-scratch model, the method does not depend on RejuvenateFormer's specific architecture; any adequate sequence-to-sequence transformer can act as the vowel restorer.
  • Restoration quality improves with corpus size: on EN-DE, BLEU rises from 23.07 with 30K training pairs to 27.31 with 100K pairs, so the approach is expected to benefit from larger-scale training.
  • The method is language-specific as presented: the removed character set is exactly the five English vowels in upper and lower case, so applying it to other languages would require redefining that set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A fair end-to-end comparison would count the restoration model's parameters, and arguably its inference compute, as part of the compressed representation; under that accounting the reported 11.4 to 13.4 times ratios would shrink, and the comparison against truly lossless codecs would be less favorable.
  • The same pattern, delete a predictable character class, compress, then regenerate, could be ported to diacritics in Arabic or Hebrew, to punctuation, or to other high-redundancy signals, with the restoration difficulty set by how much the deleted class constrains the original text.
  • BLEU scores likely overstate recoverability in this setting because fluent, plausible text can score well without matching the original sentence exactly; exact-match rate or character error rate would be a stricter test of restoration fidelity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a text-compression pipeline in which vowels are first removed from English text, the vowel-less text is compressed with the Lempel-Ziv-Welch (LZW) algorithm, and a transformer-based model (RejuvenateFormer or a pre-trained T5-Small) is used to restore the missing vowels. The central claim is that this pipeline achieves compression ratios of 12.57, 13.38, and 11.42 on BookCorpus, WMT14 EN-DE, and WMT14 EN-FR, respectively, and that these are state-of-the-art compared with both deep-learning and traditional compressors. The paper also reports restoration quality in terms of BLEU and BERTScore, and includes an ablation study on training-set size. The key evaluation issue is that the reported compression ratio is computed as the original text length divided by the LZW-compressed length of the vowel-removed text, without accounting for the fact that the compressed representation alone cannot reconstruct the original text.

Significance. If the reported ratios were genuine end-to-end compression ratios, the result would be a meaningful advance in neural text compression. The paper does provide a concrete experimental setup, compares several standard compressors after vowel removal, and reports restoration-quality metrics, and the corpus-size ablation in Table III is a reasonable check. However, the central claim is not supported by the metric as defined: the denominator excludes the decompression model, and the method is lossy because vowel removal is irreversible and restoration is approximate. These two omissions make the reported state-of-the-art compression ratios an artifact of the evaluation definition rather than a property of a complete compression system.

major comments (3)
  1. [Section IV.A, Eq. (1); Section V.C] The compression ratio is defined as Original Length / Compressed Length, but the compressed length is the LZW representation of the vowel-removed text. This is not the compressed size of the original text: the original cannot be recovered from the LZW output alone, because vowel removal is irreversible and the transformer model is required to reconstruct the missing characters. The model size is never counted in the compressed representation. A concrete estimate shows the impact: for BookCorpus, the reported compressed size is roughly 6.7M / 12.57 bytes, or about 4.26 Mb, while RejuvenateFormer has 63.23M parameters, which is about 2.02 Gb at 32 bits per parameter, so including the model makes the effective end-to-end compression ratio less than 1. A fair metric must either include the decompressor's cost or evaluate rate-distortion against the achieved reconstruction quality.
  2. [Section V.D.1, Table II] The comparison to lossless compressors is not apples-to-apples. The proposed pipeline is lossy: vowels are removed, and the transformer restores them only approximately, as shown by BLEU scores of 27.31, 25.78, and 50.45 and F1 scores around 0.89 to 0.95 in Table I. In contrast, LZMA, GZIP, ZLIB, arithmetic coding, and the GPT-based and TRACE baselines preserve the original text exactly. Reporting a higher compression ratio for a lossy scheme without a rate-distortion comparison, and calling it a state-of-the-art compression ratio, is misleading. The manuscript should either compare against lossy methods on the same terms or clearly present the result as a lossy rate-distortion trade-off.
  3. [Abstract; Section IV.B.1] The manuscript repeatedly describes the method as lossless compression, specifically in the Abstract ('a lossless compression method') and in Section IV.B.1 where LZW is described as a lossless algorithm. While LZW itself is lossless, the complete pipeline is lossy because vowel removal is not invertible and the transformer restoration is not exact. This terminology obscures the central issue: the compressed representation stores a vowel-less text that does not determine the original text. The paper should consistently refer to the approach as lossy text compression with learned restoration, and should state the reconstruction error explicitly as part of the compression claim.
minor comments (5)
  1. [Abstract] The abstract writes 'RejuvenateForme' once, while the rest of the paper uses 'RejuvenateFormer'; please fix the typo.
  2. [Table II] The column header 'GLIB' appears to be a typo for 'ZLIB'.
  3. [Table I] The BBFNMT row reports BLEU scores from the original machine-translation task, while the other rows report vowel-restoration BLEU scores; these are not directly comparable and the table should state this limitation.
  4. [Section V.D.2] The text attributes T5-Small's advantage to 'training on a larger corpus', but T5-Small is pre-trained on C4 rather than on the datasets used here; the explanation is unclear and should be revised.
  5. [Section IV.A, Eq. (2)] The notation in Eq. (2) is ambiguous: the decoder input $D_{t-1}^{out}$ and the weight matrices $W_E, W_D$ are not defined in the text. Please define all symbols used in the equation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central compression and restoration results are empirical measurements against external corpora; the reported compression-ratio metric is questionable on validity grounds, not circularity.

full rationale

The paper's derivation chain consists of a deterministic preprocessing transform (vowel removal followed by LZW compression/decompression, Eq. 1) and a standard transformer training objective (Eq. 2). The compression ratios in Table II are direct measurements of a length ratio under the paper's own definition in Section V.C, not values fitted to or logically entailed by that definition. The BLEU and BERTScore results in Table I are measured on held-out test splits of external corpora (BookCorpus, WMT14 EN-DE, EN-FR). No parameter is fitted to one subset of data and then reported as a prediction of a closely related quantity. The only self-citations ([17], [32]) support peripheral observations (LSTM classification performance and the general benefit of larger training corpora); they are not load-bearing for the central compression or restoration claims. The substantive concern with the paper is validity, not circularity: the claimed state-of-the-art compression ratio omits the decompression model's size and is computed on a lossy, vowel-removed representation, so it is not directly comparable with the lossless GPT-based and traditional baselines. This is a flaw in the evaluation metric, but it is not a step in which a claimed result is equivalent to its inputs by construction. Therefore no circularity is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No explicit fitted constants are introduced. The transformer hyperparameters (hidden size 512, 6 layers, learning rate 5e-5) are standard choices and do not directly enter the compression-ratio calculation. No new physical or mathematical entities are introduced; RejuvenateFormer is a standard transformer architecture.

assumptions (4)
  • domain assumption Removing all vowels from English text leaves enough information for a transformer to restore them with acceptable fidelity.
    Invoked in Section IV.B.2 and evaluated in Section V.D; central to the lossy restoration claim.
  • ad hoc to paper The compression ratio may be defined as original length divided by compressed length of the vowel-removed text, excluding the decompression model.
    This definition appears in Section IV.A and Section V.D; it is the basis of the state-of-the-art claim.
  • domain assumption English vowels are exactly the set of a, e, i, o, u in both upper and lower case.
    Defined in Section III.B.
  • ad hoc to paper A 100K-pair training set is sufficient to learn vowel restoration.
    Stated in Section V.A and evaluated in the ablation study of Table III; no justification for the sufficiency is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Enhanced Text Compression Approach Using Transformer-based Language Models." pith.science (2026). https://pith.science/paper/UKN4CYQG

@misc{pith2026241215250,
  author       = {Pith},
  title        = {Pith review of: An Enhanced Text Compression Approach Using Transformer-based Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UKN4CYQG}},
  note         = {Machine review of arXiv:2412.15250}
}
read the original abstract

Text compression shrinks textual data while keeping crucial information, eradicating constraints on storage, bandwidth, and computational efficacy. The integration of lossless compression techniques with transformer-based text decompression has received negligible attention, despite the increasing volume of English text data in communication. The primary barrier in advancing text compression and restoration involves optimizing transformer-based approaches with efficient pre-processing and integrating lossless compression algorithms, that remained unresolved in the prior attempts. Here, we propose a transformer-based method named RejuvenateForme for text decompression, addressing prior issues by harnessing a new pre-processing technique and a lossless compression method. Our meticulous pre-processing technique incorporating the Lempel-Ziv-Welch algorithm achieves compression ratios of 12.57, 13.38, and 11.42 on the BookCorpus, EN-DE, and EN-FR corpora, thus showing state-of-the-art compression ratios compared to other deep learning and traditional approaches. Furthermore, the RejuvenateForme achieves a BLEU score of 27.31, 25.78, and 50.45 on the EN-DE, EN-FR, and BookCorpus corpora, showcasing its comprehensive efficacy. In contrast, the pre-trained T5-Small exhibits better performance over prior state-of-the-art models.

Figures

Figures reproduced from arXiv: 2412.15250 by the authors.

Figure 1
Figure 1. (Top) Each corpus undergoes vowel removal, and the compression ratio is calculated using the compressed representation. The text is then reverted to its earliest form without vowels. (Bottom) After tokenization, the RejuvenateFormer is trained on each corpus, to proficiently generate expected outcomes. for enhancing the obtained representation. The output of each encoder layer is propagated sequentially to K identic… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. M+: Extending MemoryLLM with Scalable Long-Term Memory

    cs.CL 2025-02 conditional novelty 5.0 of 10

    M+ adds a co-trained retriever and CPU-stored long-term memory to MemoryLLM, extending tested knowledge retention from under 20k to over 160k tokens at similar GPU memory cost.

Reference graph

Works this paper leans on

33 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Worldwide texting statistics

    V . S. S. H. Office, “Worldwide texting statistics.” https://shso.vermont.gov/sites/ghsp/files/documents/Worldwide [Online; accessed 2024-07-01]

    work page 2024

  2. [2]

    100 document management statistics for the digital era

    pdfreaderpro, “100 document management statistics for the digital era.” https://www.pdfreaderpro.com/blog/document-management-statistics, 8

    work page

  3. [3]

    Lzw data compression,

    M. R. Nelson, “Lzw data compression,” Dr. Dobb’s Journal , vol. 14, no. 10, pp. 29–36, 1989

    work page 1989

  4. [4]

    Dynamic huffman coding,

    D. E. Knuth, “Dynamic huffman coding,” Journal of algorithms, vol. 6, no. 2, pp. 163–180, 1985

    work page 1985

  5. [5]

    Crossword: A Semantic Approach to Data Compression via Masking

    M. Li, R. Jin, L. Xiang, K. Shen, and S. Cui, “Crossword: A se- mantic approach to data compression via masking,” arXiv preprint arXiv:2304.01106, 2023

    work page Pith review arXiv 2023

  6. [6]

    Llmzip: Lossless text compression using large language models,

    C. S. K. Valmeekam, K. Narayanan, D. Kalathil, J.-F. Chamberland, and S. Shakkottai, “Llmzip: Lossless text compression using large language models,” arXiv preprint arXiv:2306.04050 , 2023

    arXiv 2023

  7. [7]

    Text compression-aided transformer encoding,

    Z. Li, Z. Zhang, H. Zhao, R. Wang, K. Chen, M. Utiyama, and E. Sumita, “Text compression-aided transformer encoding,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 44, no. 7, pp. 3840– 3857, 2021

    work page 2021

  8. [8]

    Explicit sentence compression for neural machine translation,

    Z. Li, R. Wang, K. Chen, M. Utiyama, E. Sumita, Z. Zhang, and H. Zhao, “Explicit sentence compression for neural machine translation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 8311–8318, 2020

    work page 2020

Show all 33 references
  1. [9]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685 , 2021

    2021 arXiv

  2. [10]

    In-context autoencoder for context compression in a large language model,

    T. Ge, J. Hu, X. Wang, S.-Q. Chen, and F. Wei, “In-context autoencoder for context compression in a large language model,” arXiv preprint arXiv:2307.06945, 2023

    2023 arXiv

  3. [11]

    Trace: A fast transformer- based general-purpose lossless compressor,

    Y . Mao, Y . Cui, T.-W. Kuo, and C. J. Xue, “Trace: A fast transformer- based general-purpose lossless compressor,” in Proceedings of the ACM Web Conference 2022, pp. 1829–1838, 2022

    2022

  4. [12]

    Approximating human- like few-shot learning with gpt-based compression,

    C. Huang, Y . Xie, Z. Jiang, J. Lin, and M. Li, “Approximating human- like few-shot learning with gpt-based compression,” arXiv preprint arXiv:2308.06942, 2023

    2023 arXiv

  5. [13]

    Lossless accelera- tion for seq2seq generation with aggressive decoding,

    T. Ge, H. Xia, X. Sun, S.-Q. Chen, and F. Wei, “Lossless accelera- tion for seq2seq generation with aggressive decoding,” arXiv preprint arXiv:2205.10350, 2022

    2022 arXiv

  6. [14]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” 2023

    2023

  7. [15]

    Tokenization is more than compression,

    C. W. Schmidt, V . Reddy, H. Zhang, A. Alameddine, O. Uzan, Y . Pinter, and C. Tanner, “Tokenization is more than compression,” arXiv preprint arXiv:2402.18376, 2024

    2024 arXiv

  8. [16]

    Deepzip: Lossless data compression using recurrent neural networks,

    M. Goyal, K. Tatwawadi, S. Chandak, and I. Ochoa, “Deepzip: Lossless data compression using recurrent neural networks,” arXiv preprint arXiv:1811.08162, 2018

    2018 arXiv

  9. [17]

    Bangla social media cyberbullying detection using deep learning,

    A. T. Rodela, H.-H. Nguyen, D. M. Farid, and M. N. Huda, “Bangla social media cyberbullying detection using deep learning,” in Interna- tional Conference on Intelligent Systems and Data Science, pp. 170–184, Springer, 2023

    2023

  10. [18]

    Texshape: Information theoretic sentence embedding for language models,

    H. K. Kale, H. Esfahanizadeh, N. Elias, O. Baser, M. Medard, and S. Vishwanath, “Texshape: Information theoretic sentence embedding for language models,” arXiv preprint arXiv:2402.05132 , 2024

    2024 arXiv

  11. [19]

    Dc-graph: A chunk optimization model based on document classification and graph learning,

    G. Zhang, X. Li, and H. Zhang, “Dc-graph: A chunk optimization model based on document classification and graph learning,” Artificial Intelligence Review, 2024

    2024

  12. [20]

    Natural-language text compression using reverse multi-delimiter codes,

    A. Anisimov, I. Zavadskyi, and T. Chudakov, “Natural-language text compression using reverse multi-delimiter codes,” Cybernetics and Sys- tems Analysis, pp. 1–12, 2024

    2024

  13. [21]

    Graph neural network and ner-based text summarization,

    I. Z. Khan, A. A. Sheikh, and U. Sinha, “Graph neural network and ner-based text summarization,” arXiv preprint arXiv:2402.05126 , 2024

    2024 arXiv

  14. [22]

    A new method for short text compres- sion,

    M. Aslany ¨urek and A. Mesut, “A new method for short text compres- sion,” IEEE Access, 2023

    2023

  15. [23]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  16. [24]

    Findings of the 2014 workshop on statistical machine translation,

    O. Bojar, C. Buck, C. Federmann, B. Haddow, P. Koehn, J. Leveling, C. Monz, P. Pecina, M. Post, H. Saint-Amand, et al., “Findings of the 2014 workshop on statistical machine translation,” in Proceedings of the ninth workshop on statistical machine translation , pp. 12–58, 2014

    2014

  17. [25]

    Aligning books and movies: Towards story-like visual explanations by watching movies and reading books,

    Y . Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler, “Aligning books and movies: Towards story-like visual explanations by watching movies and reading books,” in Proceedings of the IEEE international conference on computer vision , pp. 19–27, 2015

    2015

  18. [26]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” The Journal of Machine Learning Research, vol. 21, no. 1, pp. 5485–5551, 2020

    2020

  19. [27]

    Bertscore: Evaluating text generation with bert,

    T. Zhang*, V . Kishore*, F. Wu*, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,” in International Con- ference on Learning Representations , 2020

    2020

  20. [28]

    A call for clarity in reporting BLEU scores,

    M. Post, “A call for clarity in reporting BLEU scores,” in Proceedings of the Third Conference on Machine Translation: Research Papers , (Belgium, Brussels), pp. 186–191, Association for Computational Lin- guistics, Oct. 2018

    2018

  21. [29]

    Gzip file format specification version 4.3,

    P. Deutsch, “Gzip file format specification version 4.3,” tech. rep., Association for Computing Machinery, 1996

    1996

  22. [30]

    Zlib compressed data format specification version 3.3,

    P. Deutsch and J.-L. Gailly, “Zlib compressed data format specification version 3.3,” tech. rep., Association for Computing Machinery, 1996

    1996

  23. [31]

    Arithmetic coding for data compression,

    I. H. Witten, R. M. Neal, and J. G. Cleary, “Arithmetic coding for data compression,” Communications of the ACM, vol. 30, no. 6, pp. 520–540, 1987

    1987

  24. [32]

    Advancing bangla punctuation restoration by a monolin- gual transformer-based method and a large-scale corpus,

    M. H. Bijoy, M. F. A. Faria, M. E. Sobhani, T. Ferdoush, and S. Shatabda, “Advancing bangla punctuation restoration by a monolin- gual transformer-based method and a large-scale corpus,” in Proceedings of the First Workshop on Bangla Language Processing (BLP-2023) , pp. 18–25, 2023

    2023

  25. [2023]

    [Online; accessed 2024-07-01]

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.