Pith. sign in

REVIEW 5 major objections 5 minor 9 references

Accuracy and Consumption analysis from a compressed model by CompactifAI from Multiverse Computing

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Compressing Llama 3.1 8B with tensor networks cuts inference energy by up to 39% while keeping Ragas accuracy scores nearly identical.

desk verdict Honest but thin third-party benchmark: the energy numbers are internally consistent and worth a look, but 'accuracy maintained' rests on one unvalidated run against ChatGPT-4o references. read the letter →

arxiv 2507.08836 v1 pith:FSKCTYKH submitted 2025-07-07 cs.LG cs.PF

Damien Fovet , Shashank Chamoli , Sarah Oury , Srishti Singhal This is my paper · ORCID
classification cs.LGcs.PF
keywords modelcompressiontensornetworkslargelanguagemodelsinferenceenergycarbonfootprintRagasevaluationCodequestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a controlled comparison of a tensor-network compression method, CompactifAI, applied to Llama 3.1 8B. On a 104-question question-answering benchmark, the compressed model consumed 30.04% less total energy for short (200-token) outputs and 39.09% less for long (1000-token) outputs, with CO2 emissions reduced by roughly the same margin. Across six Ragas metrics, the compressed model scored within a small margin of the full model, slightly higher on ROUGE, BLEU, semantic similarity, and answer correctness and slightly lower on factual correctness and response relevancy. The paper concludes that compression delivers about a one-third reduction in inference energy and carbon impact without a practically meaningful loss in answer quality.

What carries the argument

CompactifAI is a compression method that uses quantum-inspired tensor networks to reduce the number of model parameters. The argument is carried by two measurement instruments and one decomposition identity: CodeCarbon supplies per-component energy and CO2 estimates, Ragas supplies six automated quality scores, and the savings identity splits total energy reduction into per-token efficiency and token-count reduction. Concretely, the compressed model consumes 25.60% less energy per generated token and outputs 18.11% fewer tokens on average, so the expected combined saving is $1 - (1 - 0.2560)(1 - 0.1811) \approx 39.08\%$, which matches the directly observed 39.09% total-energy reduction for the 1000-token configuration. This decomposition is what ties the energy measurements to the compression mechanism.

What would settle it

Replace the ChatGPT-4o reference answers with human-written gold answers for the same 104 questions and recompute the Ragas metrics for both models; if the compressed model's factual correctness or answer correctness scores fall substantially below the full model's on human references, the claim of maintained accuracy would be refuted. Independently, measure wall-plug power with a hardware wattmeter instead of CodeCarbon's proxy estimates; if the 30-39% total energy gap disappears, the efficiency claim would collapse.

Watch

Extended reading notes

Core claim

The paper's central claim is that CompactifAI compression of Llama 3.1 8B preserves answer quality while substantially cutting inference cost. Using CodeCarbon for machine-level energy tracking and Ragas for accuracy scoring against ChatGPT-4o-generated references, the authors measured total energy reductions of 30.04% (200-token configuration) and 39.09% (1000-token configuration), with GPU energy down 43.55% and 50.5%. The authors decompose these savings into two effects: the compressed model uses 25.60% less energy per generated token and generates on average 18.11% fewer tokens for the same questions, and combining these factors predicts a 39.08% reduction, closely matching the measured 39.09%. They interpret this as evidence that the efficiency gain is real and that the compressed model matches the full model's usefulness on the tested task.

Load-bearing premise

The conclusion that compression keeps accuracy intact depends on treating ChatGPT-4o-generated answers, scored by Ragas, as a valid and sensitive measure of answer quality; the paper never validates those references against human judgment, so a systematic bias in the reference set would make the small compressed-versus-full differences uninterpretable.

Editorial extensions

If this is right

  • Deploying CompactifAI-style compressed Llama 3.1 8B for question answering cuts inference energy by 30-39% and CO2 emissions by the same margin, with no meaningful change in Ragas-measured answer quality.
  • The relative savings grow with output length (39.09% for 1000-token versus 30.04% for 200-token), so compression pays off more for long-generation workloads like summarization or code generation.
  • The decomposition into per-token efficiency (-25.6%) and token-count reduction (-18.1%) gives operators a practical rule of thumb: compressed models save energy both by drawing less power per token and by answering more concisely.
  • GPU energy, the dominant consumer, drops by 43.6-50.5%, suggesting that compression can reduce the number of GPUs needed for a given inference load.
  • If reproducible across models and frameworks, tensor-network compression becomes a viable eco-design lever for frugal AI deployments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The accuracy conclusion rests entirely on ChatGPT-4o-generated references scored by Ragas; if those references carry a systematic bias shared by the tested models, the near-equality of scores could mask real quality differences, and a human-annotated ground-truth set would be a sharper test.
  • The paper measures only PyTorch on one virtual machine with a single GPU, so the energy ratios may shift under different serving stacks or hardware even if the per-token efficiency trend persists.
  • The paper does not test the vendor's claim that larger models gain proportionally more from compression; applying the same protocol to a larger Llama variant would check whether the 30-39% savings scale.
  • Because the energy cost of the compression step itself is not accounted for, the net lifecycle benefit remains unknown; measuring that upfront cost would tell practitioners how many inference calls are needed to amortize it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper reports an empirical comparison between the full Llama 3.1 8B model and a version compressed with Multiverse Computing's CompactifAI method, evaluated on 104 question-answering prompts under two generation configurations (200 and 1000 maximum tokens). The authors measure energy consumption and CO2 emissions with CodeCarbon and evaluate answer quality with six Ragas metrics (ROUGE, BLEU, semantic similarity, factual correctness, answer correctness, and response relevancy), using ChatGPT-4o-generated responses as reference ground truth. The central claims are that the compressed model reduces total energy consumption by about 30% (200-token configuration) and 39% (1000-token configuration), with comparable accuracy to the full model.

Significance. If the accuracy claim were established, the paper would provide a practically useful data point for deploying CompactifAI-compressed Llama 3.1 8B in energy-constrained settings. The energy measurement approach, including the per-token regression analysis and the decomposition of total savings into per-token efficiency and shorter outputs, is a constructive way to interpret the results. The authors also explicitly acknowledge several limitations, including the lack of comparisons with other frameworks and the absence of amortization analysis for the compression cost itself. However, the central claim that accuracy is 'maintained' is not statistically supported by the presented evidence: all accuracy comparisons rest on a single run per configuration, with no error bars, no repeated trials, and no significance testing, and the reference answers are generated by an LLM without human validation. The energy results, while plausible and large in magnitude, similarly lack repeated-measurement variance. The paper's value is thus currently as a preliminary industry benchmark rather than a definitive evaluation.

major comments (5)
  1. [§4.2, Tables/Figures 6-11] The accuracy claim is not statistically supported. Each reported Ragas metric is a single value from one run per configuration, and the differences between the compressed and full models are as small as 0.2 percentage points (semantic similarity, 200-token set: 0.7775 vs 0.7752) and at most about 4.5 points (factual correctness, 200-token set: 0.5732 vs 0.6072). Without repeated runs, confidence intervals, or paired significance tests, the statement that accuracy is 'almost equivalent' cannot be distinguished from run-to-run noise. The authors should provide at least three to five independent runs per configuration and report means with standard deviations, or otherwise justify why a single run is sufficient.
  2. [§3.2.2 and §4.2] The ground truth for all Ragas metrics is generated by ChatGPT-4o, with no human validation or comparison against an alternate reference source. If ChatGPT-4o shares systematic stylistic or factual biases with the evaluated Llama-family models, the metric gaps between the compressed and full models are biased in the same direction, making the 'maintained accuracy' conclusion uninterpretable. The paper should validate a sample of the reference answers against human judgments, or at minimum compare results using a second independently generated reference set, and report agreement between reference sources.
  3. [§3.1.3, §4.1.3] The generation configuration is internally inconsistent: both configurations specify "do_sample": "False" while also setting temperature, top_k, and top_p, and the paper does not state which inference framework served the responses or whether generation terminated due to the max_new_tokens limit. Section 4.1.3 states that in the 200-token configuration 'most of the responses were cut off in their generation,' which means the measured duration and energy for the 200-token runs include truncated generation and may reflect framework-specific batching or early-stopping behavior rather than pure model cost. The authors should clarify the sampling settings actually used, the serving framework, and the proportion of truncated responses in each configuration.
  4. [§4.1.3 and §4.2.1-4.2.2] Because the compressed model generates on average 18.11% fewer tokens, the ROUGE and BLEU advantages favoring the compressed model may largely reflect response-length effects rather than preserved or improved answer quality. Similarly, the total energy reduction of 39.09% is decomposed into a per-token reduction of 25.60% plus the 18.11% token-count reduction, but the accuracy metrics are not length-controlled. The authors should report length-normalized ROUGE/BLEU scores or otherwise analyze whether the lexical-overlap improvements persist when responses are truncated to equal length.
  5. [§4.2.4] The reporting of factual correctness is ambiguous: the text states that factual correctness was 'calculated exclusively for context-based questions,' yet the listed scores (e.g., 0.6072 for the full model, 200-token set) are presented in the same format as the overall metrics. If the reported numbers refer only to a context-based subset, the comparison with other metrics on the full 104-question set is not apples-to-apples; if they refer to the full set, the sentence is misleading. The authors should clarify the exact subset used and report the number of questions in that subset.
minor comments (5)
  1. [§3.2.2] The name 'ChatGTP 4o' is a typo; it should be 'ChatGPT-4o'.
  2. [§4.1.3] The phrase 'we can see on the Figure 1 2' is unclear; it likely refers to two figures, Figures 1 and 2, and should be reworded.
  3. [§3.1.3] In the configuration JSON, "do_sample": "False" is written as a string rather than a boolean; this should be corrected to avoid ambiguity about the actual decoding parameters.
  4. [§4.2] The paper does not state whether the 104-question dataset is publicly available or whether the evaluation code will be released; providing these would improve reproducibility.
  5. [§5] The limitations section is candid, but it does not mention the absence of multiple runs or human validation of the reference answers, which are the most consequential limitations for the accuracy claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported energy and accuracy results are direct measurements, and the token-based energy decomposition is a post-hoc consistency check, not a fitted prediction.

full rationale

This paper is an empirical measurement study rather than a derivation. The two headline claims—energy/CO2 reductions of 30.04% and 39.09%, and nearly equivalent Ragas accuracy—are directly measured and reported in Sections 4.1.2, 4.1.3, and 4.2.1–4.2.6. The only fitted quantity is the linear regression coefficient relating the number of generated tokens to energy consumption (Figure 4). This is used descriptively: the paper combines the per-token energy reduction (-25.60%) with the average token-count reduction (18.11%) and notes that the product (39.08%) is 'very close from what we observed in the previous part Total Energy Consumed (39,09%).' This is a post-hoc decomposition of an independently measured total, not a prediction that feeds back into the claim; the identity total energy = tokens × energy per token makes the match a consistency check rather than independent evidence, but it is not circular because the total reduction was already measured directly. No self-citations are load-bearing: references [2] and [3] describe the CompactifAI method but are not used to justify the measured outcomes, and the vendor statement quoted in Section 5 is explicitly framed as an untested claim. The limitations listed in Section 5—only PyTorch, only one compressed model, no amortization of compression cost—concern external validity, not circular derivation. The accuracy evaluation's reliance on ChatGPT-4o-generated reference answers is a measurement-validity concern, not a definitional circularity: the Ragas scores are computed from generated responses and do not determine the experimental inputs. Verdict: no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new theoretical entities or free parameters. All assumptions are experimental or measurement-related: the reliability of CodeCarbon, the validity of ChatGPT-4o ground truth, the meaningfulness of Ragas metrics, the representativeness of the dataset, and the stability of VM measurements. No fitted parameter is needed for the central energy-reduction claim, which is directly measured.

assumptions (5)
  • domain assumption CodeCarbon's energy and CO2 estimates accurately reflect the consumption of the test hardware
    Section 3.2.1 states CodeCarbon uses proxies for CPU and RAM when direct tracking is unavailable; the accuracy of these proxies is assumed.
  • domain assumption ChatGPT-4o-generated answers are a valid ground truth for evaluating model accuracy
    Section 3.2.2 says ChatGPT 4o was used to generate reference responses; no human validation is provided.
  • domain assumption Ragas metrics measure the intended quality dimensions (lexical, semantic, factual, relevancy)
    Section 3.2.2 lists six Ragas metrics and treats them as meaningful accuracy measures without external validation.
  • domain assumption The 104 self-designed questions are representative of QA tasks across the five listed domains
    Section 3.1.2 describes the dataset as carefully curated, but no sources or selection criteria are given.
  • domain assumption Machine-level monitoring on a shared OVH virtual machine with no user-visible parallel tasks yields stable energy measurements
    Section 3.1.1 describes the VM and the precaution to avoid parallel tasks; potential interference from other VMs on the physical host is not addressed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Accuracy and Consumption analysis from a compressed model by CompactifAI from Multiverse Computing." pith.science (2026). https://pith.science/paper/FSKCTYKH

@misc{pith2026250708836,
  author       = {Pith},
  title        = {Pith review of: Accuracy and Consumption analysis from a compressed model by CompactifAI from Multiverse Computing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FSKCTYKH}},
  note         = {Machine review of arXiv:2507.08836}
}
read the original abstract

This study evaluates the performance of a compression method, called CompactifAI, developed by Multiverse Computing, applied to the large language model Llama 3.1 8B\cite{llama}. The evaluation focused on model efficiency (in terms of energy consumption) and accuracy using respectively the frameworks Codecarbon\cite{codecarbon} and Ragas\cite{ragas}. A comparison was performed between the model compressed with CompactifAI\cite{compactifai}\cite{compactifai2} and its full-size version. Our findings reveal that the compressed model using CompactifAI not only significantly reduced the computational resources but also maintained the model accuracy, making the model more efficient, scalable and cost-effective.

Figures

Figures reproduced from arXiv: 2507.08836 by the authors.

Figure 1
Figure 1. Total carbon emissions across tests 4.1.3 Energy Consumption Total CPU Energy: • The compressed model consumed 1.29E-02 kWh for 200-tokens configuration and 2.94E-02 kWh for 1000-tokens configuration; • The full-size model consumed 1.36E-02 kWh for 200-tokens configuration and 3.60E-02 kWh for 1000-tokens configuration. The compressed model consumes less CPU energy 5.79% (200-tokens), 18.15% (1000-tokens). Total GPU… view at source ↗
Figure 2
Figure 2. Total energy consumption broken down by component across tests [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Dispatch of energy consumption broken down by component across tests 7 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Linear regression curve and experimental points for each of the 4 tests We can see the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Comparison of the 2 models in terms of consumption per token generated more pronounced for longer responses (1000-token configuration), reinforcing the effectiveness of com￾pression in extended tasks. It leads to reduction in hardware strain and financial savings, maki…
Figure 6
Figure 6. Figure 6: 9 [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 6
Figure 6. Figure 6: Comparison of ROUGE Score across tests 4.2.2 BLEU Score (Grammatical Integrity and Fluency) • BLEU (200 tokens): Compressed Model (0.1776) >Full-size Model (0.1321) • BLEU (1000 tokens): Compressed Model (0.1774) >Full-size Model (0.1319) [PITH_FULL_IMAGE:figures/full…
Figure 7
Figure 7. Figure 7: Comparison of BLEU Score across tests The compressed model exhibit improved syntactic coherence.These results indicate that the compressed model produce better responses with grammatical precision and improved structural fluency. See [PITH_FULL_IMAGE:figures/full_fig_…
Figure 8
Figure 8. Figure 8: Comparison of Semantic Similarity Score across tests There was a negligible variance observed across compression conditions. Even with the size reduction of the model, semantic coherence remains stable. See [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Comparison of Factual Correctness Score across tests 4.2.5 Answer Correctness Score • Answer Correctness Score (200 tokens): Compressed Model (200 tokens) (0.5579) >Full-size Model (200 tokens) (0.5530) • Answer Correctness Score (1000 tokens): Compressed Model (1000 t…
Figure 10
Figure 10. Figure 10: Comparison of Answer Correctness Score across tests According to the score the compressed model perform slightly better. Accuracy is not substantially reduced by compression. See [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Comparison of Response Relevancy Score across tests The full-size model handle longer contextual dependencies more effectively, preserving better align￾ment with user queries but the difference is not significant in comparison to compressed models. According to the an…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 7 canonical work pages

  1. [1]

    https://multiversecomputing.com/

    Multiverse Computing: A company focused on developing quantum computing solutions for var- ious industries. https://multiversecomputing.com/

    work page

  2. [2]

    https:// multiversecomputing.com/compactifai

    The AI model compressor to make AI systems faster, cheaper and energy efficient. https:// multiversecomputing.com/compactifai

    work page

  3. [3]

    CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networks : https://arxiv.org/abs/2401.14109

    arXiv

  4. [4]

    (Under LLAMA 3.1 community license agreement https://huggingface.co/meta-llama/Llama-3.1-8B/blob/ main/LICENSE) · Hugging Face

    A large language model developed by Meta AI: meta-llama/Llama-3.1-8B. (Under LLAMA 3.1 community license agreement https://huggingface.co/meta-llama/Llama-3.1-8B/blob/ main/LICENSE) · Hugging Face. (2024, December 6). Retrieved April 3, 2025, from https:// huggingface.co/meta-llama/Llama-3.1-8B

    work page 2024

  5. [5]

    ROUGE: A Package for Automatic Evaluation of Summaries : https://aclanthology.org/ W04-1013.pdf

    work page

  6. [6]

    3115/1073083.1073135

    BLEU: a method for automatic evaluation of machine translation : https://dl.acm.org/doi/10. 3115/1073083.1073135

    arXiv

  7. [7]

    https://docs.ragas.io/en/stable/concepts/ metrics/available_metrics/semantic_similarity/

    Semantic similarity from Ragas documentation. https://docs.ragas.io/en/stable/concepts/ metrics/available_metrics/semantic_similarity/

    work page

  8. [8]

    CodeCarbon (Under MIT license) — CodeCarbon 2.8.3 documentation

    mlco2. CodeCarbon (Under MIT license) — CodeCarbon 2.8.3 documentation. (n.d.). Retrieved April 3, 2025, from https://mlco2.github.io/codecarbon/

    work page 2025

Show all 9 references
  1. [9]

    Ragas (n.d.)

    Exploding Gradients. Ragas (n.d.). (Under Apache 2.0 license). Retrieved April 3, 2025, from https://docs.ragas.io/en/latest/. 14

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.