Pith. sign in

REVIEW 5 cited by

$\text{Memory}^3$: Language Modeling with Explicit Memory

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.01178 v1 pith:TEJA2DYE submitted 2024-07-01 cs.CL cs.AIcs.LG

Hongkang Yang , Zehao Lin , Wenjin Wang , Hao Wu , Zhiyu Li , Bo Tang , Wenqiang Wei , Jinbo Wang
show 8 more authors
Zeyun Tang Shichao Song Chenyang Xi Yu Yu Kai Chen Feiyu Xiong Linpeng Tang Weinan E
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords memoryexplicitknowledgellmscostmodeltextinference
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

The training and inference of large language models (LLMs) are together a costly process that transports knowledge from raw data to meaningful computation. Inspired by the memory hierarchy of the human brain, we reduce this cost by equipping LLMs with explicit memory, a memory format cheaper than model parameters and text retrieval-augmented generation (RAG). Conceptually, with most of its knowledge externalized to explicit memories, the LLM can enjoy a smaller parameter size, training cost, and inference cost, all proportional to the amount of remaining "abstract knowledge". As a preliminary proof of concept, we train from scratch a 2.4B LLM, which achieves better performance than much larger LLMs as well as RAG models, and maintains higher decoding speed than RAG. The model is named $\text{Memory}^3$, since explicit memory is the third form of memory in LLMs after implicit memory (model parameters) and working memory (context key-values). We introduce a memory circuitry theory to support the externalization of knowledge, and present novel techniques including a memory sparsification mechanism that makes storage tractable and a two-stage pretraining scheme that facilitates memory formation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MemSFT: Mitigating Alignment Tax with an External Parametric Memory

    cs.LG 2026-07 conditional novelty 6.0 of 10

    MemSFT attaches a retriever-imitating 8B memory plus a word-level router to frozen Qwen3 backbones, boosting domain scores by ~36 points while holding general-benchmark averages essentially flat, where full SFT loses ...

  2. Learn to Memorize: Optimizing LLM-based Agents with Adaptive Memory Framework

    cs.LG 2025-08 conditional novelty 5.0 of 10

    A learnable memory cycle with adaptive retrieval, merging, and storage, trained online, improves LLM agent accuracy on HotpotQA and MemDaily for most backbones.

  3. MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Models

    cs.CL 2025-05 reject novelty 4.0 of 10

    A unified memory-operating-system design for LLMs, built around a MemCube abstraction, is presented without any experimental validation.

  4. Toward Efficient Agents: Memory, Tool learning, and Planning

    cs.AI 2026-01 conditional novelty 3.0 of 10

    A survey that organizes efficiency techniques for LLM agents into memory, tool learning, and planning, and consolidates benchmarks and metrics for measuring cost-performance trade-offs.

  5. Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures

    cs.LG 2025-08 unverdicted novelty 3.0 of 10

    Memory-augmented Transformer research is organized into a three-axis taxonomy bridging neuroscience memory concepts to network designs, but no new result is produced.

Pith tools