Pith. sign in

REVIEW 3 major objections 6 minor 77 references

Addressing the sustainable AI trilemma: a case study on LLM agents and RAG

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Memory-augmented LLM agents and RAG pipelines run at 1-2% of optimal energy efficiency, with LLM-based indexing costing about 10 joules per token versus 0.01 joules for vector indexing.

desk verdict Useful energy measurements for RAG/agent memory design, but the equity claim rests on a confounded hardware comparison; referee it with revisions requested. read the letter →

arxiv 2501.08262 v1 pith:CR7IUPWG submitted 2025-01-14 cs.CY

Hui Wu , Xiaoyang Wang , Zhong Fan This is my paper · ORCID
classification cs.CY
keywords sustainableAILLMagentsretrieval-augmentedgenerationenergyefficiencyinferencedigitalequityenvironmentalsustainabilitymemorymodules
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the way LLM agents and retrieval-augmented generation are currently built is unsustainable: the memory modules that make them useful also make them extremely energy-expensive. It introduces the Sustainable AI Trilemma, the claim that capability, digital equity, and environmental sustainability cannot all be maximized at once, and uses the memory module as a case study. The paper measures the energy cost of each memory operation and proposes new ratios, such as the energy-to-relevance ratio and the Generation Energy Optimality Ratio ($\mathrm{GEOR} = E_{\mathrm{optimal}}/E_{\mathrm{real}}$), to make the trade-offs visible. The central empirical finding is that even when an agent answers correctly, it consumes roughly 50 to 100 times the theoretical minimum energy, and that adding standard optimization steps such as query expansion or compression multiplies energy use by 15 to 46 times over a plain retrieve-and-generate baseline.

What carries the argument

The argument is carried by an analytic energy model for LLM inference, $E = \alpha_0 T_{\mathrm{in}} + \alpha_1 T_{\mathrm{out}} + \alpha_2 T_{\mathrm{in}} T_{\mathrm{out}}$, with coefficients fitted to a quantized Llama-3.1-8B model ($\alpha_0 = 0.0429$, $\alpha_1 = 9.109$, $\alpha_2 = 0.000513$, $R^2 = 0.989$). The paper decomposes a memory-augmented agent into six LLM operations (memory formation, retrieval necessity detection, query optimization, reranking, compression, and generation) and assigns each an energy formula built on that model. On top of this it defines ratios that tie energy to effectiveness: RERR and EAR for memory formation, GERR, EFR, and ECR for generation, and GEOR, which compares real energy to the minimal energy needed for a correct answer. These ratios turn raw watt measurements into the paper's efficiency claims.

What would settle it

Run the same memory-formation and generation pipeline with the same quantized Llama-3.1-8B model on the A100 cluster and on the RTX 4080, holding precision and model fixed; if the energy-per-token gap and the steeper top-K scaling largely disappear, then the resource-scarcity explanation for the poverty trap is unsupported. Alternatively, run the full-precision 8B model on the RTX 4080 to isolate the quantization effect.

Watch

Extended reading notes

Core claim

The paper's central claim is that the dominant design pattern in LLM agents, offloading memory formation, retrieval decisions, query rewriting, reranking, and compression to the LLM itself, is energetically wasteful. Concretely, LLM-based indexing methods (keyword, summary, graph) consume around $10^1$ J per token, while simple vector indexing consumes around $10^{-2}$ J per token, a thousand-fold difference. When all memory-reading and memory-utilization operations are enabled, total energy rises to 15–46 times the plain retrieve-and-generate baseline (Table 5), and the Generation Energy Optimality Ratio, the ratio of the theoretical minimum energy for a correct answer to the real energy spent, is only 1.012–1.456% across three datasets (Table 6). The paper also claims that resource-constrained deployments suffer disproportionately: a quantized 8B model on a consumer GPU pays about 35% more energy per token than the same model at full precision on an A100 cluster, and its energy penalty grows faster with retrieval depth, a pattern the authors call a 'technological poverty trap.'

Load-bearing premise

The claim that resource-constrained environments face a disproportionate efficiency penalty rests on comparing a quantized 8B model on a consumer GPU with full-precision 8B and 70B models on A100 clusters, so hardware, precision, and model size change at once; if most of the penalty comes from quantization overhead or consumer-GPU memory bandwidth rather than from resource scarcity itself, the 'technological poverty trap' conclusion would not follow.

Editorial extensions

If this is right

  • If the measurements hold, defaulting to vector-based (non-LLM) indexing for memory formation would cut indexing energy by roughly three orders of magnitude with little loss in retrieval relevance or accuracy.
  • Retrieval necessity detection is the one optimization step that is consistently cheap (1.17–1.67 times the baseline), so deploying it before expensive operations like query expansion or compression would avoid much of the wasted energy.
  • Query optimization and compression should be applied selectively: they multiply energy by roughly 12–35 times and 5–17 times respectively, and on simple conversational tasks query optimization can even hurt answer quality.
  • The LLM-centric agent design, which delegates every subtask to the LLM, should be replaced by hybrid pipelines that use traditional algorithms for indexing, retrieval, and reranking, reserving LLM calls for generation.
  • The 1–2% GEOR figures imply that current systems have roughly a 50–100 times energy headroom for improvement, so efficiency gains of an order of magnitude are possible without changing the model's answers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own data suggest a direct test of the poverty trap: running the same quantized 8B model on the A100 cluster would separate quantization overhead from hardware scarcity, and if the 35% penalty vanishes, the trilemma's access dimension needs a different explanation.
  • The GEOR metric could be extended beyond memory modules to planning and tool-use loops, where the same pattern of many small LLM calls may produce similar inefficiencies.
  • Because GEOR is defined relative to a theoretical minimum, it implicitly recommends a target: systems should be engineered to approach the minimal data and operation set, which points to better retrieval precision (fewer irrelevant memories) as a primary lever.
  • A policy implication the paper leaves implicit is that sustainability labels for LLM services would need standardized energy accounting of the kind these metrics provide; otherwise the evaluation dilemma blocks progress on the other two.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a 'Sustainable AI Trilemma' (capability, equity, sustainability) and uses a case study of memory modules in LLM agents and retrieval-augmented generation (RAG) to quantify inference energy costs. It designs three experiments: (Task 1) comparing vector, keyword, summary, and graph indexing on internal and external memory datasets; (Task 2) measuring the incremental energy of retrieval detection, query optimization, reranking, compression, and generation, leading to new efficiency metrics (RERR, EAR, GEOR, GERR/EFR/ECR); and (Task 3) comparing a resource-constrained stack (quantized Llama-3.1-8B on an RTX 4080) with a resource-abundant stack (full-precision 8B/70B on A100s). The headline findings are that LLM-based indexing costs roughly 10 J per token versus 1e-2 J for vector indexing, that adding optimization operations multiplies energy by 15-46x over a retrieve-and-generate baseline, that correctly answered queries run at only 1-2% of the theoretical optimal energy (GEOR), and that resource-constrained deployments suffer disproportionate penalties.

Significance. If the measurements hold, the paper provides a valuable empirical benchmark: the order-of-magnitude gap between non-LLM and LLM-based indexing in Task 1 and the multiplicative overheads in Table 5 are directly measured and likely robust. The proposed metrics are simple and reusable, and the availability of code is a strength. The conceptual framing of the trilemma is useful for discussing equity and sustainability trade-offs. However, the paper's equity-oriented claim, the 'technological poverty trap', rests on a confounded comparison in Section 4.4 and is not supported by the current data, and the absence of variance/uncertainty reporting prevents strong quantitative conclusions about the smaller differences. The paper is therefore a promising empirical contribution whose central efficiency results are credible, but whose broader policy claims need to be either backed by a controlled experiment or substantially softened.

major comments (3)
  1. [§4.4, Figures 4b and 5] The resource-constrained vs. resource-abundant comparison changes four factors at once: hardware (RTX 4080 vs. eight A100s), precision (8-bit vs. full), model size (8B vs. 8B/70B), and serving stack (local LlamaIndex vs. NVIDIA NIM). The observed 35% higher energy per memory token and the steeper top-K scaling in Figure 4b cannot be attributed to resource scarcity alone, so the 'technological poverty trap' conclusion in Section 5 is not established by this experiment. I request a controlled comparison in which one factor is varied at a time, or a rewrite that limits the claim to a description of the specific bundles.
  2. [Tables 2-6; §4.1] All quantitative results are reported as averages over ten repetitions without variance, confidence intervals, or significance tests, and the measurement protocol in Section 4.1 does not state whether idle/background power was subtracted or how the CPU and GPU readings were combined. These omissions are load-bearing for the 35% difference in Section 4.4 and for the 15-46x multiples in Table 5, since those ratios could change materially with protocol choice and run-to-run variability; please add error bars, statistical tests where relevant, and a precise energy measurement protocol.
  3. [§4.4, Model Size Scaling Benefits] The text says the 70B model achieves 'performance improvements outpace energy cost increases', but the reported numbers show energy per memory token rising by a factor of about 2.68 (67.5 vs. 25.2 J) while RERR and EAR rise only by factors of about 2.19 and 2.01, respectively; this implies relevance and accuracy gains of only roughly 22-34% against a 168% energy increase. The correct interpretation is that efficiency degrades sublinearly with model size, not that performance improvements outpace energy costs, and this should be corrected because it feeds the 'Cost-Effectiveness Paradox' narrative.
minor comments (6)
  1. [§3.3.2, Eqs. (8)-(10)] Equations (8)-(10) use the labels GRER, 'F ER', and CER, while the text and tables use GERR, EFR, and ECR; please align the notation.
  2. [§4.1] The paper states that 'we sampled some data' but does not report the sampling method or seeds; please specify how the subsets listed in Table 1 were drawn.
  3. [§4.3b] The paper does not define how 'correctly answered queries' were identified; please state the criterion (e.g., exact match, ROUGE threshold, or LLM judge).
  4. [Tables 2 and 3] Tables 2 and 3 mix decimal formats (e.g., '10.23e-01' and '8.15e-01'); please standardize the scientific notation.
  5. [§4.3b] The fitted energy-model coefficients are reported without standard errors; since GEOR depends on them, please include uncertainties or confidence intervals.
  6. [Figure 5] Figure 5 shows averages of the three LLM-based indexing methods; the spread across Keyword, Summary, and Graph is not visible, so consider plotting the individual methods or adding error bars.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's headline energy comparisons are direct measurements, and the imported energy model is refit locally and used as a modeling component rather than as a prediction of the target finding.

full rationale

The derivation chain does not reduce to its own inputs. The main quantitative claims are empirically measured: vector indexing energy per token (10^-2 J) versus LLM-based indexing (10^1 J), the 15x-46x energy multiples of adding RAG operations, and the resource-constrained versus resource-abundant comparisons all rest on direct power measurements via powercap and pynvml. The energy model E = alpha_0*Tin + alpha_1*Tout + alpha_2*Tin*Tout is imported from prior work [27] and explicitly refit to local measurements in Section 4.3 ("we derived the coefficients alpha_K,0 = 0.042933, alpha_K,1 = 9.109322, and alpha_K,2 = 0.000513... The high R-squared value of 0.989 validates that this model effectively captures the energy-token numbers relationship"). This is standard modeling, not circularity: the fitted model is used to estimate the ideal-cost numerator Eg in GEOR, while Ereal is measured directly. The novel metrics (RERR, EAR, GERR, EFR, ECR, GEOR) are explicitly defined ratios, not empirical discoveries that are then used to justify themselves. No load-bearing step is justified by a self-citation: the authors do not cite their own prior work to support the trilemma framework or the energy model. The confounded hardware comparison in Section 4.4 is a threat to the internal validity of the 'technological poverty trap' conclusion, but that is a correctness or experimental-design concern, not circularity. The paper's central results are self-contained against external measurements and benchmarks, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central analysis rests on a small set of imported assumptions: the bilinear energy model from Wilkins et al. [27], the validity of GPT-4-based quality scores, and the availability of ground-truth memories and answers for the GEOR optimal-case computation. Three coefficients of the energy model are fitted to local measurements, and top-K is a hand-chosen experimental setting. No new physical entities are introduced.

free parameters (4)
  • α0 (input token energy coefficient) = 0.042933 J/token
    Fitted by OLS to measured Llama-3.1-8B-Instruct (8-bit) energy over 100 (input length, output length) pairs; used in Eq. (3) and in the GEOR calculation.
  • α1 (output token energy coefficient) = 9.109322 J/token
    Fitted along with α0 and α2 in Section 4.3b; output generation dominates energy, consistent with prior work.
  • α2 (interaction coefficient) = 0.000513 J/token^2
    Interaction term from the prior energy model, fit to the same OLS regression.
  • retrieval top-K = 5
    Hand-chosen retrieval depth used in all Tasks; affects energy and quality scores.
assumptions (4)
  • domain assumption LLM inference energy is bilinear in input and output token counts: E = α0 Tin + α1 Tout + α2 Tin Tout (Eq. 3).
    Taken from Wilkins et al. [27] and assumed to hold for Llama-3.1 models including the quantized 8B variant; the paper only fits the coefficients, not the functional form.
  • domain assumption GPT-4-based UpTrain scores are valid measures of Relevance, Faithfulness, and Correctness.
    Used for RERR, EAR, GERR, EFR, ECR and for selecting 'correct' generations in the GEOR analysis; no validation of the scorer itself is provided.
  • domain assumption Ground-truth memories MG and answers AG are available for computing the optimal-case energy Eoptimal.
    The GEOR definition (Eqs. 6-7) requires knowing exactly which memories and answer are correct; this is only true in evaluation datasets, limiting real-world interpretation.
  • ad hoc to paper The Sustainable AI Trilemma (capability, equity, sustainability) is the right interpretive lens for the measurements.
    The conceptual framework is proposed in Section 2.1 and used to structure the discussion; it is not derived from the data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Addressing the sustainable AI trilemma: a case study on LLM agents and RAG." pith.science (2026). https://pith.science/paper/CR7IUPWG

@misc{pith2026250108262,
  author       = {Pith},
  title        = {Pith review of: Addressing the sustainable AI trilemma: a case study on LLM agents and RAG},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CR7IUPWG}},
  note         = {Machine review of arXiv:2501.08262}
}
read the original abstract

Large language models (LLMs) have demonstrated significant capabilities, but their widespread deployment and more advanced applications raise critical sustainability challenges, particularly in inference energy consumption. We propose the concept of the Sustainable AI Trilemma, highlighting the tensions between AI capability, digital equity, and environmental sustainability. Through a systematic case study of LLM agents and retrieval-augmented generation (RAG), we analyze the energy costs embedded in memory module designs and introduce novel metrics to quantify the trade-offs between energy consumption and system performance. Our experimental results reveal significant energy inefficiencies in current memory-augmented frameworks and demonstrate that resource-constrained environments face disproportionate efficiency penalties. Our findings challenge the prevailing LLM-centric paradigm in agent design and provide practical insights for developing more sustainable AI systems.

Figures

Figures reproduced from arXiv: 2501.08262 by the authors.

Figure 1
Figure 1. The Sustainable AI Trilemma. Operational Dilemma: Scalability vs. Sustainability Scaling AI services to meet increasing demand exacerbates the tension between scalability and sustainability. Expanding the number of deployed LLM-based agents requires main￾taining consistent Quality of Service (QoS) under growing workloads. However, the infrastructure required to ensure QoS—such as low-latency inference, real-time res… view at source ↗
Figure 2
Figure 2. Design of Memory Modules in LLM Agents or RAG. The figure contains the operations that require LLM [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. New efficiency metrics to balance energy cost and task performance. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The Relationship between Energy Cost Multiple and Memory Reading Workload (top-K value) [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Comparison of Memory Formation in Resource-constrained vs. Resource-abundant Environments [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 23 canonical work pages

  1. [1]

    The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink

    David Patterson et al. “The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink”. In: Computer 55.7 (2022), pp. 18–28. DOI: 10.1109/MC.2022.3148714

    arXiv 2022

  2. [2]

    Reducing the Carbon Impact of Generative AI Inference (today and in 2035)

    Andrew A Chien et al. “Reducing the Carbon Impact of Generative AI Inference (today and in 2035)”. In: Proceedings of the 2nd Workshop on Sustainable Computer Systems . HotCarbon ’23. Boston, MA, USA: Association for Computing Machinery, 2023. DOI: 10.1145/3604930.3605705

    arXiv 2023

  3. [3]

    Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning

    Radosvet Desislavov, Fernando Martínez-Plumed, and José Hernández-Orallo. “Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning”. In: Sustainable Computing: Infor- matics and Systems 38 (2023), p. 100857. ISSN : 2210-5379. DOI: https://doi.org/10.1016/j.suscom. 2023.100857

    arXiv 2023

  4. [4]

    Preventing the Immense Increase in the Life-Cycle Energy and Carbon Footprints of LLM-Powered Intelligent Chatbots

    Peng Jiang, Christian Sonne, Wangliang Li, Fengqi You, and Siming You. “Preventing the Immense Increase in the Life-Cycle Energy and Carbon Footprints of LLM-Powered Intelligent Chatbots”. In: Engineering 40 (2024), pp. 202–210. ISSN : 2095-8099. DOI: https://doi.org/10.1016/j.eng.2024.04.002

    doi:10.1016/j.eng.2024.04.002 2024

  5. [5]

    Triple Bottom Line or Trilemma? Global Tradeoffs Between Prosperity, Inequality, and the Environment

    Tong Wu et al. “Triple Bottom Line or Trilemma? Global Tradeoffs Between Prosperity, Inequality, and the Environment”. In: World Development 178 (2024), p. 106595

    work page 2024

  6. [6]

    The Rise and Potential of Large Language Model Based Agents: A Survey

    Zhiheng Xi et al. "The Rise and Potential of Large Language Model Based Agents: A Survey". 2023. arXiv: 2309.07864 [cs.AI]

    arXiv 2023

  7. [7]

    Chatgpt needs spade (sustainability, privacy, digital divide, and ethics) evaluation: A review

    Sunder Ali Khowaja, Parus Khuwaja, Kapal Dev, Weizheng Wang, and Lewis Nkenyereye. “Chatgpt needs spade (sustainability, privacy, digital divide, and ethics) evaluation: A review”. In:Cognitive Computation (2024), pp. 1–23. DOI: 10.1007/s12559-024-10285-1 . 15 A PREPRINT - JANUARY 15, 2025

    work page doi:10.1007/s12559-024-10285-1 2024

  8. [8]

    LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Models

    Ahmad Faiz et al. "LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Models". 2024. arXiv: 2309.14393 [cs.CL]

    arXiv 2024

Show all 77 references
  1. [9]

    The AI trilemma: Saving the planet without ruining our jobs

    Ekkehard Ernst. “The AI trilemma: Saving the planet without ruining our jobs”. In: Frontiers in Artificial Intelligence 5 (2022). ISSN : 2624-8212. DOI: 10.3389/frai.2022.886561

    2022

  2. [10]

    Challenging AI for Sustainability: what ought it mean?

    Sophia Falk and Aimee van Wynsberghe. “Challenging AI for Sustainability: what ought it mean?” In:AI and Ethics 4.4 (2024), pp. 1345–1355

    2024

  3. [11]

    Survey on AI Sustainability: Emerging Trends on Learning Algorithms and Research Challenges

    Zhenghua Chen, Min Wu, Alvin Chan, Xiaoli Li, and Yew-Soon Ong. “Survey on AI Sustainability: Emerging Trends on Learning Algorithms and Research Challenges”. In: IEEE Computational Intelligence Magazine 18.2 (2023), pp. 60–77. DOI: 10.1109/MCI.2023.3245733

    2023

  4. [12]

    Quantifying the Carbon Emissions of Machine Learning

    Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. "Quantifying the Carbon Emissions of Machine Learning". 2019. arXiv: 1910.09700 [cs.CY]

    2019 arXiv

  5. [13]

    Green Algorithms: Quantifying the Carbon Footprint of Computation

    Loïc Lannelongue, Jason Grealey, and Michael Inouye. “Green Algorithms: Quantifying the Carbon Footprint of Computation”. In: Advanced Science 8.12 (2021), p. 2100707. DOI: https://doi.org/10.1002/advs. 202100707

    2021 doi

  6. [14]

    Agent design pattern catalogue: A collection of architectural patterns for foundation model based agents

    Yue Liu et al. “Agent design pattern catalogue: A collection of architectural patterns for foundation model based agents”. In: Journal of Systems and Software 220 (2025), p. 112278. ISSN : 0164-1212. DOI: 10.1016/j.jss. 2024.112278

    2025

  7. [15]

    Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud

    Himel Ghosh. "Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud"

  8. [16]

    A Joint Study of the Challenges, Opportunities, and Roadmap of MLOps and AIOps: A Systematic Survey

    Josu Diaz-de-Arcaya, Ana I. Torre-Bastida, Gorka Zárate, Raúl Miñón, and Aitor Almeida. “A Joint Study of the Challenges, Opportunities, and Roadmap of MLOps and AIOps: A Systematic Survey”. In: ACM Comput. Surv. 56.4 (Oct. 2023). ISSN : 0360-0300. DOI: 10.1145/3625289

    2023 doi

  9. [17]

    Whose ChatGPT? Unveiling Real-World Educational Inequalities Introduced by Large Language Models

    Renzhe Yu, Zhen Xu, Sky CH-Wang, and Richard Arum. "Whose ChatGPT? Unveiling Real-World Educational Inequalities Introduced by Large Language Models". 2024. arXiv: 2410.22282 [cs.CY]

    2024 arXiv

  10. [18]

    Examining Potential Harms of Large Language Models (LLMs) in Africa

    Rehema Baguma, Hajarah Namuwaya, Joyce Nakatumba-Nabende, and Qazi Mamunur Rashid. “Examining Potential Harms of Large Language Models (LLMs) in Africa”. In: International Conference on Safe, Secure, Ethical, Responsible Technologies and Emerging Applications. Springer. 2023, pp. 3–19

    2023

  11. [19]

    Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects

    Yuheng Cheng et al. "Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects". 2024. arXiv: 2401.03428 [cs.AI]

    2024 arXiv

  12. [20]

    Understanding the planning of LLM agents: A survey

    Xu Huang et al. "Understanding the planning of LLM agents: A survey". 2024. arXiv: 2402.02716 [cs.AI]

    2024 arXiv

  13. [21]

    LLM With Tools: A Survey

    Zhuocheng Shen. "LLM With Tools: A Survey". 2024. arXiv: 2409.18807 [cs.AI]

    2024 arXiv

  14. [22]

    Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely

    Siyun Zhao et al. "Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely". 2024. arXiv: 2409.14924 [cs.CL]

    2024 arXiv

  15. [23]

    A Survey on the Memory Mechanism of Large Language Model based Agents

    Zeyu Zhang et al. "A Survey on the Memory Mechanism of Large Language Model based Agents". 2024. arXiv: 2404.13501 [cs.AI]

    2024 arXiv

  16. [24]

    Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models

    Yue Zhang et al. "Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models". 2023. arXiv: 2309.01219 [cs.CL]

    2023 arXiv

  17. [25]

    Sustainable LLM Serving: Environmental Implications, Challenges, and Opportunities : Invited Paper

    Yi Ding and Tianyao Shi. “Sustainable LLM Serving: Environmental Implications, Challenges, and Opportunities : Invited Paper”. In: 2024 IEEE 15th International Green and Sustainable Computing Conference (IGSC). 2024, pp. 37–38. DOI: 10.1109/IGSC64514.2024.00016

    2024

  18. [26]

    From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference

    Siddharth Samsi et al. “From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference”. In: 2023 IEEE High Performance Extreme Computing Conference (HPEC). 2023, pp. 1–9. DOI: 10.1109/HPEC58863.2023.10363447

    2023

  19. [27]

    Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems

    Grant Wilkins, Srinivasan Keshav, and Richard Mortier. "Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems". 2024. arXiv: 2407.04014 [cs.DC]

    2024 arXiv

  20. [28]

    Measuring and Improving the Energy Efficiency of Large Language Models Inference

    Mauricio Fadel Argerich and Marta Patiño-Martínez. “Measuring and Improving the Energy Efficiency of Large Language Models Inference”. In: IEEE Access 12 (2024), pp. 80194–80207. DOI: 10.1109/ACCESS.2024. 3409745

    2024 doi

  21. [29]

    Learn to Code Sustainably: An Empirical Study on LLM-based Green Code Generation

    Tina Vartziotis et al. "Learn to Code Sustainably: An Empirical Study on LLM-based Green Code Generation"

  22. [30]

    Method and evaluations of the effective gain of artificial intelligence models for reducing CO2 emissions

    Paul Delanoë, Dieudonné Tchuente, and Guillaume Colin. “Method and evaluations of the effective gain of artificial intelligence models for reducing CO2 emissions”. In: Journal of Environmental Management 331 (2023), p. 117261. ISSN : 0301-4797. DOI: https://doi.org/10.1016/j.j...

    2023

  23. [31]

    arXiv: 2403.03344 [cs.SE]

    arXiv

  24. [32]

    MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversa- tion

    Junru Lu et al. "MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversa- tion". 2023. arXiv: 2308.08239 [cs.CL]

    2023 arXiv

  25. [33]

    When Large Language Models Meet Vector Databases: A Survey

    Zhi Jing, Yongye Su, and Yikun Han. "When Large Language Models Meet Vector Databases: A Survey". 2024. arXiv: 2402.01763 [cs.DB]. 16 A PREPRINT - JANUARY 15, 2025

    2024 arXiv

  26. [34]

    Enhancing Large Language Model with Self-Controlled Memory Framework

    Bing Wang et al. "Enhancing Large Language Model with Self-Controlled Memory Framework". 2024. arXiv: 2304.13343 [cs.CL]

    2024 arXiv

  27. [35]

    Memorybank: Enhancing large language models with long-term memory

    Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. “Memorybank: Enhancing large language models with long-term memory”. In: Proceedings of the AAAI Conference on Artificial Intelligence. V ol. 38. 2024, pp. 19724–19731

    2024

  28. [36]

    Prompted LLMs as Chatbot Modules for Long Open-domain Conversation

    Gibbeum Lee, V olker Hartmann, Jongho Park, Dimitris Papailiopoulos, and Kangwook Lee. “Prompted LLMs as Chatbot Modules for Long Open-domain Conversation”. In: Findings of the Association for Computational Linguistics: ACL 2023 . Ed. by Anna Rogers, Jordan Boyd-Graber, and Na...

    2023 doi

  29. [37]

    MemGPT: Towards LLMs as Operating Systems

    Charles Packer et al. "MemGPT: Towards LLMs as Operating Systems". 2024. arXiv: 2310.08560 [cs.AI]

    2024 arXiv

  30. [38]

    RET-LLM: Towards a General Read-Write Memory for Large Language Models

    Ali Modarressi, Ayyoob Imani, Mohsen Fayyaz, and Hinrich Schütze. "RET-LLM: Towards a General Read-Write Memory for Large Language Models". 2024. arXiv: 2305.14322 [cs.CL]

    2024 arXiv

  31. [39]

    Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memory

    Lei Liu et al. "Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memory". 2023. arXiv: 2311.08719 [cs.CL]

    2023 arXiv

  32. [40]

    Retrieve Only When It Needs: Adaptive Retrieval Augmentation for Hallucination Mitigation in Large Language Models

    Hanxing Ding, Liang Pang, Zihao Wei, Huawei Shen, and Xueqi Cheng. "Retrieve Only When It Needs: Adaptive Retrieval Augmentation for Hallucination Mitigation in Large Language Models". 2024. arXiv:2402.10612 [cs.CL]

    2024

  33. [41]

    Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation

    Ruiyang Ren et al. "Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation". 2024. arXiv: 2307.11019 [cs.CL]

    2024 arXiv

  34. [42]

    Least-to-Most Prompting Enables Complex Reasoning in Large Language Models

    Denny Zhou et al. "Least-to-Most Prompting Enables Complex Reasoning in Large Language Models". 2023. arXiv: 2205.10625 [cs.AI]

    2023 arXiv

  35. [43]

    Self-Knowledge Guided Retrieval Augmentation for Large Language Models

    Yile Wang, Peng Li, Maosong Sun, and Yang Liu. “Self-Knowledge Guided Retrieval Augmentation for Large Language Models”. In: Findings of the Association for Computational Linguistics: EMNLP 2023 . Ed. by Houda Bouamor, Juan Pino, and Kalika Bali. Singapore: Association for Com...

    2023 doi

  36. [44]

    Measuring and Narrowing the Compositionality Gap in Language Models

    Ofir Press et al. “Measuring and Narrowing the Compositionality Gap in Language Models”. In: Findings of the Association for Computational Linguistics: EMNLP 2023. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 5687–5711. DOI: 10.18653/v1/2023.findings-emnlp.378

    2023 doi

  37. [45]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

    Jason Wei et al. "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models". 2023. arXiv: 2201.11903 [cs.CL]

    2023 arXiv

  38. [46]

    Precise Zero-Shot Dense Retrieval without Relevance Labels

    Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. “Precise Zero-Shot Dense Retrieval without Relevance Labels”. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Ed. by Anna Rogers, Jordan Boyd-Graber, and Na...

    2023 doi

  39. [47]

    LangChain MultiQueryRetriever Documentation

    LangChain. "LangChain MultiQueryRetriever Documentation". Accessed: 2025-01-10. 2024. URL: https: / / python . langchain . com / v0 . 1 / docs / modules / data _ connection / retrievers / MultiQueryRetriever/

    2025

  40. [48]

    Query Rewriting in Retrieval-Augmented Large Language Models

    Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. “Query Rewriting in Retrieval-Augmented Large Language Models”. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics, Dec. 2...

    2023

  41. [49]

    Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models

    Huaixiu Steven Zheng et al. "Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models"

  42. [50]

    arXiv: 2310.06117 [cs.LG]

    arXiv

  43. [51]

    Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking

    Shengyao Zhuang, Bing Liu, Bevan Koopman, and Guido Zuccon. "Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking". 2023. arXiv: 2310.13243 [cs.IR]

    2023 arXiv

  44. [52]

    The probabilistic relevance framework: BM25 and beyond

    Stephen Robertson and Hugo Zaragoza. “The probabilistic relevance framework: BM25 and beyond”. In: Foundations and Trends in Information Retrieval3.4 (2009), pp. 333–389

    2009

  45. [53]

    Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

    Nils Reimers and Iryna Gurevych. "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks"

  46. [54]

    Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents

    Weiwei Sun et al. “Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents”. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Ed. by Houda Bouamor, Juan Pino, and Kalika Bali. Singapore: Association for Com...

    2023 doi

  47. [55]

    Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting

    Zhen Qin et al. “Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting”. In: Findings of the Association for Computational Linguistics: NAACL 2024. Ed. by Kevin Duh, Helena Gomez, and Steven Bethard. Mexico City, Mexico: Association for Computational...

    2024 doi

  48. [56]

    Zero-Shot Listwise Document Reranking with a Large Language Model

    Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. "Zero-Shot Listwise Document Reranking with a Large Language Model". 2023. arXiv: 2305.02156 [cs.IR]. 17 A PREPRINT - JANUARY 15, 2025

    2023 arXiv

  49. [57]

    Improving Passage Retrieval with Zero-Shot Question Generation

    Devendra Sachan et al. “Improving Passage Retrieval with Zero-Shot Question Generation”. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 3781–...

    2022 doi

  50. [58]

    RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation

    Fangyuan Xu, Weijia Shi, and Eunsol Choi. "RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation". 2023. arXiv: 2310.04408 [cs.CL]

    2023 arXiv

  51. [59]

    Compressing Context to Enhance Inference Efficiency of Large Language Models

    Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. "Compressing Context to Enhance Inference Efficiency of Large Language Models". 2023. arXiv: 2310.06201 [cs.CL]

    2023 arXiv

  52. [60]

    Holistic Evaluation of Language Models

    Percy Liang et al. "Holistic Evaluation of Language Models". 2023. arXiv: 2211.09110 [cs.CL]

    2023 arXiv

  53. [61]

    PRCA: Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual Adapter

    Haoyan Yang et al. "PRCA: Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual Adapter". 2023. arXiv: 2310.18347 [cs.CL]

    2023 arXiv

  54. [62]

    Evaluation of Retrieval-Augmented Generation: A Survey

    Hao Yu et al. "Evaluation of Retrieval-Augmented Generation: A Survey". 2024. arXiv:2405.07437 [cs.CL]

    2024 arXiv

  55. [63]

    Evaluating Very Long-Term Conversational Memory of LLM Agents

    Adyasha Maharana et al. "Evaluating Very Long-Term Conversational Memory of LLM Agents". 2024. arXiv: 2402.17753 [cs.CL]

    2024 arXiv

  56. [64]

    LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. "LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models". 2023. arXiv: 2310.05736 [cs.CL]

    2023 arXiv

  57. [65]

    LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compres- sion

    Zhuoshi Pan et al. "LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compres- sion". 2024. arXiv: 2403.12968 [cs.CL]

    2024 arXiv

  58. [66]

    Ai arxiv dataset

    James Calam. "Ai arxiv dataset". Accessed: 2025-01-10. 2023. URL: https://huggingface.co/datasets/ jamescalam/ai-arxiv

    2025

  59. [67]

    ARAGOG: Advanced RAG Output Grading

    Matouš Eibich, Shivay Nagpal, and Alexander Fred-Ojala. "ARAGOG: Advanced RAG Output Grading". 2024. arXiv: 2404.01037 [cs.CL]

    2024 arXiv

  60. [68]

    HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

    Zhilin Yang et al. "HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering". 2018. arXiv: 1809.09600 [cs.CL]

    2018 arXiv

  61. [69]

    MuSiQue: Multihop Questions via Single-hop Question Composition

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. “MuSiQue: Multihop Questions via Single-hop Question Composition”. In: Transactions of the Association for Computational Linguistics 10 (2022), pp. 539–554

    2022

  62. [70]

    Powercap Linux Kernel Interface

    "Powercap Linux Kernel Interface". Accessed: 2025-01-10. 2024. URL: https://www.kernel.org/doc/ html/latest/power/powercap/powercap.html

    2025

  63. [71]

    NVIDIA Management Library (NVML) Python Bindings

    NVIDIA Corporation. "NVIDIA Management Library (NVML) Python Bindings". Accessed: 2025-01-10. 2024. URL: https://pypi.org/project/nvidia-ml-py/

    2025

  64. [72]

    "UpTrain"

    UpTrain AI. "UpTrain". Accessed: 2025-01-10. 2024. URL: https://github.com/uptrain-ai/uptrain

    2025

  65. [73]

    ROUGE: A Package for Automatic Evaluation of Summaries

    Chin-Yew Lin. “ROUGE: A Package for Automatic Evaluation of Summaries”. In:Text Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics, July 2004, pp. 74–81. URL: https : / / aclanthology.org/W04-1013/

    2004

  66. [76]

    LlamaIndex

    Jerry Liu. "LlamaIndex". Nov. 2022. URL: https://github.com/jerryjliu/llama_index

    2022

  67. [77]

    Searching for best practices in retrieval-augmented generation

    Xiaohua Wang et al. “Searching for best practices in retrieval-augmented generation”. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024, pp. 17716–17736. 18

    2024

  68. [2019]

    arXiv: 1908.10084 [cs.CL]

    1908 arXiv

  69. [2024]

    arXiv: 2411.15664 [cs.DC]

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.