Pith. sign in

REVIEW 4 major objections 5 minor 52 references

PMKLC: Parallel Multi-Knowledge Learning-based Lossless Compression for Large-Scale Genomics Database

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims PMKLC, a three-model neural compressor, achieves the best average compression ratio (1.845 bits/base), throughput, and robustness among 14 baselines on 15 genomic datasets.

desk verdict Solid engineering paper in genomic compression, but the headline CR claim rides on an unresolved accounting question: whether SPrM size is included in reported compressed sizes. read the letter →

arxiv 2507.12805 v1 pith:DNG2OMPM submitted 2025-07-17 cs.LG cs.AIcs.CLcs.DB

Hui Sun , Yanfeng Ding , Liping Yi , Huidong Ma , Gang Wang , Xiaoguang Liu , Cheng Zhong , Wentong Cai This is my paper · ORCID
classification cs.LGcs.AIcs.CLcs.DB
keywords losslesscompressiongenomicsneuralnetworkmulti-knowledgelearningGPUacceleration(sk)-merencodingparallelrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PMKLC is a lossless genomic compressor built around three neural models that contribute different kinds of knowledge: a frozen public model pretrained on many species, a frozen private model pretrained on the file being compressed, and an online dynamic model that adapts as it compresses. A model selector automatically turns the static models on or off depending on file size, and a probability mixer blends their predictions. The paper reports that on 15 real genome datasets, PMKLC-S (single GPU) averages 1.845 bits per base and PMKLC-M (multi-GPU) averages 1.854, both slightly better than the strongest learning baseline at 1.855, while multi-GPU mode runs up to 10.7 times faster than an LSTM-based compressor. If these numbers hold, PMKLC makes learning-based compression fast and memory-lean enough for routine backup of large genome databases.

What carries the argument

The central object is the Automated Multi-Knowledge Learning-based Compression Framework (AMKLCF): a three-model architecture in which a frozen Static Public Model (SPuM, pretrained on a multi-species corpus), a frozen Static Private Model (SPrM, pretrained on the target dataset), and an online Dynamic Model (DM, updated during compression) are combined by a Model Selector and a Probability Mixer. A supporting mechanism is the GPU-accelerated ($s,k$)-mer Encoder (GskE), which converts DNA substrings of length $k$ into base-4 numeric tokens with a step $s$, shrinking the data and lowering memory; the paper sets $s=k=3$ and context length $t=32$ as the default balance.

What would settle it

Compress a small genome such as PlFa with PMKLC-S in two configurations: dynamic model plus selector only, and the full AMKLCF with SPuM and SPrM forced on. If the full configuration does not produce a lower bits/base than the selector-only configuration, the paper's core claim that multi-knowledge learning boosts compression is falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims to be the first multi-GPU lossless compressor for genomic sequences, and to show that combining three knowledge sources beats any single one. SPuM supplies cross-species public knowledge, SPrM supplies file-specific global knowledge, and the Dynamic Model supplies online adaptation; the Model Selector chooses which static models to load, and the Probability Mixer learns the weight $\alpha$ that blends them. With this design, PMKLC-S reaches an average 1.845 bits/base and PMKLC-M 1.854 bits/base on 15 datasets, edging out the best learning-based baseline (1.855 bits/base) while improving average throughput to 20.987 KB/s (single GPU) and 60.897 KB/s (multi-GPU). The paper also reports the best robustness score among learning compressors (CRP 4.455% and 4.546%) and large CPU-memory savings, and concludes that PMKLC is the best overall balance of ratio, speed, robustness, and memory use.

Load-bearing premise

The load-bearing premise is that adding the frozen public and private models to the online dynamic model actually improves compression enough to justify their compute cost; if the model selector alone is what drives the gains, the multi-knowledge claim collapses.

Editorial extensions

If this is right

  • If the reported ratios hold, learning-based compressors can now edge below 1.85 bits per base on a diverse 15-genome benchmark, beating the previous best learning baseline by a small but consistent margin.
  • Multi-GPU acceleration via data chunking and stepwise model passing lifts throughput by up to 10.7x over LSTM-based compressors, making neural compression plausible for routine database backup rather than archival curiosity.
  • Automated model selection by file size keeps ratio stable across datasets from 6 MB to 1 GB, addressing the cold-start and model-overhead problems that previously made static neural compressors brittle on small files.
  • The GPU-accelerated (s,k)-mer encoder reduces CPU memory by up to 84% relative to the best learning baseline, widening the range of devices that can run these compressors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the ablation results suggest the static public and private models contribute little on small files (MODE B and C were worse than MODE A), so a simpler dynamic-model-plus-selector architecture may capture most of the benefit at lower memory cost; the paper does not test this combination directly.
  • Beyond the paper: the 500MB threshold that flips model selection is likely species- and dataset-dependent; a learned or adaptive threshold could improve cross-species robustness beyond the fixed rule reported here.
  • Beyond the paper: the stepwise model passing and block partitioning ideas are generic, so applying them to other neural compressors could transfer most of the more than 1000% throughput gain without the multi-knowledge machinery.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes PMKLC, a learning-based lossless compressor for genomic databases, built on an Automated Multi-Knowledge Learning-based Compression Framework (AMKLCF) that combines a static public model, a static private model, and a dynamic model under a Model Selector, together with a GPU-accelerated (s,k)-mer encoder and multi-GPU data partitioning with Step-wise Model Passing. Two modes are presented: PMKLC-S for a single GPU and PMKLC-M for multiple GPUs. The authors benchmark these against 14 (or 15, see comments) baselines on 15 real-world genomic datasets, reporting compression ratio, throughput, robustness, CPU/GPU memory usage, and an extensive ablation study. The headline claims are that PMKLC-S and PMKLC-M achieve the best average compression ratio (1.845 and 1.854 bits/base, respectively, vs. 1.855 for MSDLC), large throughput improvements, and the best robustness among the learning-based baselines.

Significance. If confirmed, PMKLC would be a practically useful contribution: it addresses throughput and memory limitations of prior learning-based genomic compressors, it ships with an open-source repository and dataset links, and it provides a broad comparison across 15 datasets with both traditional and learning-based baselines. The ablation study and resource measurements are also valuable. However, the central 'best compression ratio' claim currently rests on a very small average margin over MSDLC, and the paper does not state whether the required Static Private Model is included in the reported compressed size. Until this accounting question is resolved and the statistical significance of the small margins is established, the significance of the empirical claims is conditional.

major comments (4)
  1. [§4.2.3, §5.1.3, Eq. (3), Table 4] The compression ratio definition in Eq. (3) is ambiguous about whether Size_compressed includes the Static Private Model (SPrM). Section 4.2.3 states that SPrM 'needs to be preserved to losslessly restore the original data,' and Section 4.2.1 enables SPrM for datasets larger than 500 MB, which includes AcSc and TaGu in the benchmark. The paper nowhere states that Size_compressed includes SPrM. This is load-bearing because PMKLC-S's average advantage over MSDLC is only 0.010 bits/base (1.845 vs. 1.855) and 0.005 bits/base on TaGu; even a serialized SPrM checkpoint of a few MB could erase or reverse this margin. The authors must clarify whether SPrM is included and, if not, recompute all compression ratios with SPrM counted as part of the compressed output.
  2. [§5.6, Table 7, Contribution 1] The ablation study does not support the claim that the multi-knowledge framework itself 'enhances compression ratio and robustness.' On OrSa, MODE A (only the dynamic model) achieves 1.903 bits/base, while MODE B and MODE C, which add SPrM and SPuM, both achieve 1.993 bits/base—a clear degradation. Only MODE S, which includes the Model Selector that disables SPrM on small data, recovers to 1.893 bits/base. This indicates that on small datasets the Model Selector, not the multi-knowledge combination, is responsible for the compression-ratio benefit. The authors should temper the contribution statement or provide additional evidence for the multi-knowledge mechanism, for example by analyzing datasets where the static models genuinely improve the ratio despite the model-size overhead.
  3. [§5.1.4, Tables 4–6] The paper reports no variance, confidence intervals, or statistical tests for the compression ratio results. The margins over the strongest baselines are very small: PMKLC-S improves on MSDLC by 0.557% on average and is actually worse on WaMe and DrMe. The hyperparameters (s, k, t, bs, and the 500 MB threshold) were tuned on five validation datasets before being fixed for the test datasets. Given this tuning and the small margins, the benchmark-superiority claim would be considerably strengthened by repeated runs, standard errors, or a paired statistical comparison across the nine test datasets.
  4. [Abstract and §5.2, Table 4] The abstract's headline 'average compression ratio improvement up to 73.609%' is based on DNA-BiLSTM with the static model included in the compressed file, not on the best learning-based baseline. Against DNA-BiLSTM* (model excluded), the improvement is 3.274%, and against MSDLC it is 0.557%. Since the abstract reports the 73.609% figure without this caveat, it is misleading. The main text does disclose the DNA-BiLSTM* distinction, but the abstract should either state the comparison explicitly or quote a more representative improvement.
minor comments (5)
  1. [Throughout] There are several typographical errors, including 'Uiversity' in the author affiliations, 'Adptive' in Section 3.2, and 'leaning-based' in the abstract. These should be corrected.
  2. [Abstract, Table 3, Table 4] The paper claims 14 baselines (7 traditional and 7 learning-based), but Table 4 lists 8 traditional methods (Spring, NAF, LZMA2, XZ, PPMD, PBzip2, Gzip, SnZip) and 7 learning-based methods, for a total of 15. Additionally, PBzip2 appears in Table 4 but is absent from Table 3, and MFCompress and GenoZip are mentioned in the introduction but not included in the benchmark. The authors should reconcile these counts and tables.
  3. [§5.6, Table 7] The text says that 'the introduction of SMP alleviated the cold-start issue (MODE-E),' but in Table 7 row E has SMP marked as absent (✗) and MGPU marked as present (✓). Either the table columns are mislabeled or the text is incorrect; this should be clarified.
  4. [Tables 5 and 6] In Tables 5 and 6, the rows for PMKLC-S show an 'Improve-M' value and the rows for PMKLC-M show an 'Improve-S' value. These are relative comparisons between the two PMKLC modes, not against the baselines, which is confusing. The authors should separate inter-mode comparisons from baseline improvements or relabel the columns.
  5. [§5.1.3, Eq. (5)] The robustness metric CRP is defined with CR_u, which is later described as the average compression ratio, but the notation is not introduced before its use. Please define CR_u explicitly at first use.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central claims are empirical benchmark results, and no prediction reduces to a fitted parameter, a self-citation chain, or a definitional identity.

full rationale

This is an empirical systems paper rather than a derivation. The headline numbers (Tables 4-6) are measured compression ratios, throughputs, memory usages, and robustness values obtained by running PMKLC and 14 baselines on 15 datasets; none of these quantities is defined in terms of PMKLC's own outputs in a way that would make the result true by construction. Eq. (3)-(5) are standard metric definitions, and the paper does not substitute a fitted parameter for a predicted quantity. Hyperparameters (s, k, t, bs) were tuned on the separate validation/tuning datasets in Table 2 (OrSa, DiCo, EnIn, ScPo, DaRe) before reporting on the nine testing datasets in Table 4; this is conventional model selection, not a fitted-input-called-prediction pattern. The ablation in Table 7 even shows that the multi-knowledge modules can hurt compression ratio on OrSa (MODE B/C: 1.993 bits/base vs. MODE A: 1.903 bits/base), so the paper does not manufacture its 'multi-knowledge helps' narrative purely by construction. Several self-citations appear ([4], [9], [33], [41]) for design inheritance, the SMP mechanism, and baselines, but the central 'best average CR' claim is not justified by a self-citation or a uniqueness theorem; it is an executable benchmark comparison. The reviewer-raised question about whether Size_compressed in Eq. (3) includes the SPrM that section 4.2.3 says must be preserved is a measurement-accounting/verifiability concern, not a circularity: the text does not state that SPrM is excluded, and Eq. (3) does not by construction equate PMKLC's compressed size to a value omitting it. Since no circular step can be exhibited with the paper's own equations or definitions, the circularity score is low.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical entities are introduced. The free parameters are hand-tuned configuration choices; the axioms are standard coding assumptions plus domain-specific transfer claims that are not proven. The robustness claim relies on an implicit redefinition of robustness as cross-dataset variance.

free parameters (4)
  • (s,k)-mer step and window (s=3,k=3) = 3,3
    Chosen via grid search on validation datasets; Figure 6 shows other (s,k) values trade off ratio versus throughput.
  • context length t = 32
    Tuned on OrSa, EnIn, ScPo, and DaRe (Section 5.9); larger t improves ratio but reduces throughput.
  • batch size bs = 320
    Tuned on validation datasets (Section 5.8); larger batch improves throughput but worsens ratio.
  • Model Selector threshold = 500 MB
    Rule to enable SPuM for files <=500MB else SPrM; set based on experiments, not learned or derived (Sections 4.2.1 and 5.1.4).
assumptions (5)
  • standard math Entropy coding is lossless and exact.
    Assumed by the framework; the specific entropy encoder is not described (Section 4, Figure 2).
  • domain assumption Cross-species genomic similarity makes SPuM transferable.
    Justified by a human-gorilla similarity example (Section 4a); no quantitative validation of transfer for the test datasets is provided.
  • domain assumption The (s,k)-mer encoding is reversible.
    The paper does not explicitly prove reconstruction from overlapping k-mers; it relies on standard dictionary-based encoding (Section 4.1, Figure 3).
  • domain assumption The benchmark datasets and tuning sets represent real workload distribution.
    Hyperparameters are chosen via a validation set, and no evidence of distribution shift handling is given beyond variance across test datasets (Section 5.1).
  • domain assumption Robustness measured by variance across test datasets captures robustness to perturbations.
    CRP across 9 files is used as robustness; no actual probability distribution perturbations are applied (Section 5.1.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of PMKLC: Parallel Multi-Knowledge Learning-based Lossless Compression for Large-Scale Genomics Database." pith.science (2026). https://pith.science/paper/DNG2OMPM

@misc{pith2026250712805,
  author       = {Pith},
  title        = {Pith review of: PMKLC: Parallel Multi-Knowledge Learning-based Lossless Compression for Large-Scale Genomics Database},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DNG2OMPM}},
  note         = {Machine review of arXiv:2507.12805}
}
abstract

Learning-based lossless compressors play a crucial role in large-scale genomic database backup, storage, transmission, and management. However, their 1) inadequate compression ratio, 2) low compression \& decompression throughput, and 3) poor compression robustness limit their widespread adoption and application in both industry and academia. To solve those challenges, we propose a novel \underline{P}arallel \underline{M}ulti-\underline{K}nowledge \underline{L}earning-based \underline{C}ompressor (PMKLC) with four crucial designs: 1) We propose an automated multi-knowledge learning-based compression framework as compressors' backbone to enhance compression ratio and robustness; 2) we design a GPU-accelerated ($s$,$k$)-mer encoder to optimize compression throughput and computing resource usage; 3) we introduce data block partitioning and Step-wise Model Passing (SMP) mechanisms for parallel acceleration; 4) We design two compression modes PMKLC-S and PMKLC-M to meet the complex application scenarios, where the former runs on a resource-constrained single GPU and the latter is multi-GPU accelerated. We benchmark PMKLC-S/M and 14 baselines (7 traditional and 7 leaning-based) on 15 real-world datasets with different species and data sizes. Compared to baselines on the testing datasets, PMKLC-S/M achieve the average compression ratio improvement up to 73.609\% and 73.480\%, the average throughput improvement up to 3.036$\times$ and 10.710$\times$, respectively. Besides, PMKLC-S/M also achieve the best robustness and competitive memory cost, indicating its greater stability against datasets with different probability distribution perturbations, and its strong ability to run on memory-constrained devices.

Figures

Figures reproduced from arXiv: 2507.12805 by the authors.

Figure 1
Figure 1. The compression diagrams for Static (a), Adaptive (b), and Semi-adaptive (c) compression frameworks. Here, GD and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Compression pipeline of the proposed PMKLC using 2 GPUs. Here, GD, MSGD, CBF, and GskE denote Genomics Data, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The GPU-based (𝑠,𝑘)-mer Encoder (GskE) as [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Parallelization using Step-wise Model Passing (PSM) [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: The impact of different (𝑠,𝑘)-mer encoder on the GaGa dataset. “★” denotes the default configuration [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: The impact of different batch size (𝑏𝑠) and context length (𝑡) on datasets OrSa, EnIn, ScPo, and DaRe decompression throughput. In this study, we set 𝑡 = 32 to balance compression ratio and throughput; however, other values of 𝑡 are still valid and can be chosen accord…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 46 canonical work pages

  1. [1]

    Cnsa: a data repository for archiving omics data

    Xueqin Guo, Fengzhen Chen, Fei Gao, Ling Li, Ke Liu, Lijin You, Cong Hua, Fan Yang, Wanliang Liu, Chunhua Peng, et al. Cnsa: a data repository for archiving omics data. Database, 2020:baaa055, 2020

    work page 2020

  2. [2]

    A survey and benchmark evaluation for neural-network-based lossless universal compressors toward multi- source data

    Hui Sun, Huidong Ma, Feng Ling, Haonan Xie, Yongxia Sun, Liping Yi, Meng Yan, Cheng Zhong, Xiaoguang Liu, and Gang Wang. A survey and benchmark evaluation for neural-network-based lossless universal compressors toward multi- source data. Frontiers of Computer Science , 19(7):1–16, 2025

    work page 2025

  3. [3]

    Deepdna: A hybrid convo- lutional and recurrent neural network for compressing human mitochondrial genomes

    Rongjie Wang, Yang Bai, Yan-Shuo Chu, Zhenxing Wang, Yongtian Wang, Min- grui Sun, Junyi Li, Tianyi Zang, and Yadong Wang. Deepdna: A hybrid convo- lutional and recurrent neural network for compressing human mitochondrial genomes. In 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 270–274. IEEE, 2018

    work page 2018

  4. [4]

    Genomics data lossless compression with (s, k)-mer encoding and deep neural networks

    Hui Sun, Liping Yi, Huidong Ma, Yongxia Sun, Yingfeng Zheng, Wenwen Cui, Meng Yan, Gang Wang, and Xiaoguang Liu. Genomics data lossless compression with (s, k)-mer encoding and deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 12577–12585, 2025

    work page 2025

  5. [5]

    ricme: Long-read based mobile element variant detection using sequence realignment and identity calculation

    Huidong Ma, Cheng Zhong, Hui Sun, Danyang Chen, and Haixiang Lin. ricme: Long-read based mobile element variant detection using sequence realignment and identity calculation. In International Symposium on Bioinformatics Research and Applications, pages 165–177. Springer, 2023

    work page 2023

  6. [6]

    Langdon Jr

    Glen G. Langdon Jr. An introduction to arithmetic coding. IBM J. Res. Dev. , 28(2):135–149, 1984

    work page 1984

  7. [7]

    Xz official website, 2024

    XZ. Xz official website, 2024. [Online]. Available: https://github.com/tukaani- project/xz

    work page 2024

  8. [8]

    A free file archiver for extremely high compression, 2024

    Ipavlov. A free file archiver for extremely high compression, 2024. [Online]. Available: https://www.7- zip.org

    work page 2024

Show all 52 references
  1. [9]

    Msdzip: Universal lossless compression for multi-source data via stepwise-parallel and learning-based prediction

    Huidong Ma, Hui Sun, Liping Yi, Yanfeng Ding, Xiaoguang Liu, and Gang Wang. Msdzip: Universal lossless compression for multi-source data via stepwise-parallel and learning-based prediction. In Proceedings of the ACM on Web Conference 2025, pages 3543–3551, 2025

    2025

  2. [10]

    A dna sequence corpus for compression benchmark

    Diogo Pratas and Armando J Pinho. A dna sequence corpus for compression benchmark. In Practical Applications of Computational Biology and Bioinformatics, 12th International Conference, pages 208–215. Springer, 2019

    2019

  3. [11]

    Compressing genomic sequences by using deep learning

    Wenwen Cui, Zhaoyang Yu, Zhuangzhuang Liu, Gang Wang, and Xiaoguang Liu. Compressing genomic sequences by using deep learning. In International Conference on Artificial Neural Networks , pages 92–104. Springer, 2020

    2020

  4. [12]

    Geneformer: Learned gene compression using transformer-based context modeling

    Zhanbei Cui, Tongda Xu, Jia Wang, Yu Liao, and Yan Wang. Geneformer: Learned gene compression using transformer-based context modeling. In ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8035–8039. IEEE, 2024

    2024

  5. [13]

    Mfcompress: a compression tool for fasta and multi-fasta data

    Armando J Pinho and Diogo Pratas. Mfcompress: a compression tool for fasta and multi-fasta data. Bioinformatics, 30(1):117–118, 2014

    2014

  6. [14]

    Genozip: a universal extensible genomic data compressor

    Divon Lan, Ray Tobler, Yassine Souilmi, and Bastien Llamas. Genozip: a universal extensible genomic data compressor. Bioinformatics, 37(16):2225–2230, 2021

    2021

  7. [15]

    Incremental journaling backup utility and archiver, 2024

    Matt Mahoney. Incremental journaling backup utility and archiver, 2024

    2024

  8. [16]

    Lzma official website

    Igor Pavlov. Lzma official website. https://tukaani.org/lzma/, 2013

    2013

  9. [17]

    Cleary and Ian H

    John G. Cleary and Ian H. Witten. Data compression using adaptive coding and partial string matching. IEEE Trans. Commun., 32(4):396–402, 1984

    1984

  10. [18]

    Implementing the PPM data compression scheme

    Alistair Moffat. Implementing the PPM data compression scheme. IEEE Trans. Commun., 38(11):1917–1921, 1990

    1917

  11. [19]

    Cleary and W

    John G. Cleary and W. J. Teahan. Unbounded length contexts for PPM. Comput. J., 40(2/3):67–75, 1997

    1997

  12. [20]

    Trace: A fast transformer- based general-purpose lossless compressor

    Yu Mao, Yufei Cui, Tei-Wei Kuo, and Chun Jason Xue. Trace: A fast transformer- based general-purpose lossless compressor. In Proceedings of the ACM Web Conference 2022, pages 1829–1838, 2022

    2022

  13. [21]

    Faster and stronger lossless compression with optimized autoregressive framework

    Yu Mao, Jingzong Li, Yufei Cui, and Chun Jason Xue. Faster and stronger lossless compression with optimized autoregressive framework. In 2023 60th ACM/IEEE Design Automation Conference (DAC), pages 1–6. IEEE, 2023

    2023

  14. [22]

    Dzip: improved general-purpose lossless compression based on novel neural network modeling

    Mohit Goyal, Kedar Tatwawadi, Shubham Chandak, and Idoia Ochoa. Dzip: improved general-purpose lossless compression based on novel neural network modeling. In 2021 data compression conference (DCC) , pages 153–162. IEEE, 2021

    2021

  15. [23]

    Deepzip: Lossless data compression using recurrent neural networks

    Mohit Goyal, Kedar Tatwawadi, Shubham Chandak, and Idoia Ochoa. Deepzip: Lossless data compression using recurrent neural networks. arXiv preprint arXiv:1811.08162, 2018

    2018 arXiv

  16. [24]

    Long short-term memory

    Hochreiter Sepp and Schmidhuber Jürgen. Long short-term memory. Neural Computation MIT-Press, 1997

    1997

  17. [25]

    A review on the attention mecha- nism of deep learning

    Zhaoyang Niu, Guoqiang Zhong, and Hui Yu. A review on the attention mecha- nism of deep learning. Neurocomputing, 452:48–62, 2021

    2021

  18. [26]

    Rethinking learning-based method for lossless genome compression

    Han Yang, Fei Gu, and Jieping Ye. Rethinking learning-based method for lossless genome compression. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1–5. IEEE, 2023

    2023

  19. [27]

    Gencoder: A novel convolutional neural net- work based autoencoder for genomic sequence data compression

    KS Sheena and Madhu S Nair. Gencoder: A novel convolutional neural net- work based autoencoder for genomic sequence data compression. IEEE/ACM Transactions on Computational Biology and Bioinformatics , (01):1–12, 2024

    2024

  20. [28]

    Lec-codec: Learning- based genome data compression

    Zhenhao Sun, Meng Wang, Shiqi Wang, and Sam Kwong. Lec-codec: Learning- based genome data compression. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2024

    2024

  21. [29]

    Bert: Pre- training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  22. [30]

    Gomez, and Łukasz Kaiser

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, and Łukasz Kaiser. Attention is all you need. Advances in Neural Information Processing Systems , 30, 2017. KDD ’25, August 3–7, 2025, Toronto, ON, Canada. Hui Sun et al

    2017

  23. [31]

    Llmzip: Lossless text com- pression using large language models

    Chandra Shekhara Kaushik Valmeekam, Krishna Narayanan, Dileep Kalathil, Jean-Francois Chamberland, and Srinivas Shakkottai. Llmzip: Lossless text com- pression using large language models. arXiv preprint arXiv:2306.04050, 2023

    2023 arXiv

  24. [32]

    lstm-compress, 2017

    Byron Knoll. lstm-compress, 2017. https://github.com/byronknoll/lstm-compress

    2017

  25. [33]

    Adaptive lossless compression for genomics data by multiple (s, k)-mer encoding and xlstm

    Hui Sun, Yanfeng Ding, Liping Yi, Huidong Ma, Haonan Xie, Gang Wang, and Xiaoguang Liu. Adaptive lossless compression for genomics data by multiple (s, k)-mer encoding and xlstm. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICAS...

    2025

  26. [34]

    K-mer counting: memory-efficient strategy, parallel computing and field of application for bioinformatics

    Ming Xiao, Jiakun Li, Song Hong, Yongtao Yang, Junhua Li, Jianxin Wang, Jian Yang, Wenbiao Ding, and Le Zhang. K-mer counting: memory-efficient strategy, parallel computing and field of application for bioinformatics. In 2018 IEEE International Conference on Bioinformatics and...

    2018

  27. [35]

    xlstm: Extended long short-term memory

    Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksan- dra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. arXiv preprint arXiv:2405.04517, 2024

    2024 arXiv

  28. [36]

    Accelerating general- purpose lossless compression via simple and scalable parameterization

    Yu Mao, Yufei Cui, Tei-Wei Kuo, and Chun Jason Xue. Accelerating general- purpose lossless compression via simple and scalable parameterization. In Pro- ceedings of the 30th ACM International Conference on Multimedia, pages 3205–3213, 2022

    2022

  29. [37]

    Parallel algorithm for sensitive sequence recogni- tion from long-read genome data with high error rate

    ZHONG Cheng and SUN Hui. Parallel algorithm for sensitive sequence recogni- tion from long-read genome data with high error rate. Journal on Communica- tion/Tongxin Xuebao, 44(2), 2023

    2023

  30. [38]

    The ncbi biosystems database

    Lewis Y Geer, Aron Marchler-Bauer, Renata C Geer, Lianyi Han, Jane He, Siqian He, Chunlei Liu, Wenyao Shi, and Stephen H Bryant. The ncbi biosystems database. Nucleic acids research, 38(suppl_1):D492–D496, 2010

    2010

  31. [39]

    Introduction to data compression

    Khalid Sayood. Introduction to data compression . Morgan Kaufmann, 2017

    2017

  32. [40]

    A mathematical theory of communication

    Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948

    1948

  33. [41]

    Multi-source data lossless compression via parallel expansion mapping and xlstm

    Huidong Ma, Hui Sun, Liping Yi, Xiaoguang Liu, and Gang Wang. Multi-source data lossless compression via parallel expansion mapping and xlstm. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025

    2025

  34. [42]

    Jarvis3: an efficient encoder for genomic data

    Maria J P Sousa, Armando J Pinho, and Diogo Pratas. Jarvis3: an efficient encoder for genomic data. Bioinformatics, 40(12):btae725, 12 2024

    2024

  35. [43]

    Gzip official website

    Gzip. Gzip official website. http://www.gzip.org/, 1996

    1996

  36. [44]

    Snzip, a compression/decompression tool based on snappy

    Kubo Takehiro. Snzip, a compression/decompression tool based on snappy. https://github.com/kubo/snzip, 2016

    2016

  37. [45]

    Spring: a next-generation compressor for fastq data

    Shubham Chandak, Kedar Tatwawadi, Idoia Ochoa, Mikel Hernaez, and Tsachy Weissman. Spring: a next-generation compressor for fastq data. Bioinformatics, 35(15):2674–2676, 2019

    2019

  38. [46]

    Bsc official website

    Ilya Grebnov. Bsc official website. https://github.com/IlyaGrebnov/libbsc

  39. [47]

    Nucleotide archival format (naf) enables efficient lossless reference-free com- pression of dna sequences

    Kirill Kryukov, Mahoko Takahashi Ueda, So Nakagawa, and Tadashi Imanishi. Nucleotide archival format (naf) enables efficient lossless reference-free com- pression of dna sequences. Bioinformatics, 35(19):3826–3828, 2019

    2019

  40. [48]

    Zstd official website, 2024

    Zstd. Zstd official website, 2024. [Online]. Available: https://facebook.github.io/zstd/

    2024

  41. [49]

    Pmffrc: a large-scale genomic short reads compression optimizer via memory modeling and redundant clustering

    Hui Sun, Yingfeng Zheng, Haonan Xie, Huidong Ma, Xiaoguang Liu, and Gang Wang. Pmffrc: a large-scale genomic short reads compression optimizer via memory modeling and redundant clustering. BMC bioinformatics, 24(1):454, 2023

    2023

  42. [50]

    PQSDC: a parallel lossless compressor for quality scores data via sequences partition and Run-Length prediction mapping

    Hui Sun, Yingfeng Zheng, Haonan Xie, Huidong Ma, Cheng Zhong, Meng Yan, Xiaoguang Liu, and Gang Wang. PQSDC: a parallel lossless compressor for quality scores data via sequences partition and Run-Length prediction mapping. Bioinformatics, page btae323, 05 2024

    2024

  43. [51]

    Sr2c: A structurally redundant short reads collapser for optimizing dna data compression

    Hui Sun, Huidong Ma, Yingfeng Zheng, Haonan Xie, Xiaofei Wang, Xiaoguang Liu, and Gang Wang. Sr2c: A structurally redundant short reads collapser for optimizing dna data compression. In 2023 IEEE 29th International Conference on Parallel and Distributed Systems (ICPADS) , page...

    2023

  44. [52]

    Gtz: a fast compression and cloud transmission tool optimized for fastq files

    Yuting Xing, Gen Li, Zhenguo Wang, Bolun Feng, Zhuo Song, and Chengkun Wu. Gtz: a fast compression and cloud transmission tool optimized for fastq files. BMC bioinformatics, 18(16):233–242, 2017

    2017

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.