Pith. sign in

REVIEW 5 cited by

Value Residual Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.17897 v5 pith:6ZO2YXP5 submitted 2024-10-23 cs.CL

Zhanchao Zhou , Tianyi Wu , Zhiyun Jiang , Fares Obeid , Zhenzhong Lan This is my paper
classification cs.CL
keywords informationvaluecachehiddenlayerslearningperformanceresformer
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

While Transformer models have achieved remarkable success in various domains, the effectiveness of information propagation through deep networks remains a critical challenge. Standard hidden state residuals often fail to adequately preserve initial token-level information in deeper layers. This paper introduces ResFormer, a novel architecture that enhances information flow by incorporating value residual connections in addition to hidden state residuals. And a variant is SVFormer, where all layers share the first layer's value embedding. Comprehensive empirical evidence demonstrates ResFormer achieves equivalent validation loss with 16.11\% fewer model parameters and 20.3\% less training data compared to Transformer, while maintaining similar memory usage and computational cost. Besides, SVFormer reduces KV cache size by nearly half with only a small performance penalty and can be integrated with other KV-efficient methods, yielding further reductions in KV cache, with performance influenced by sequence length and cumulative learning rate.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Role-Decoupled Attention Residuals: Separating Matching and Content Retrieval Across Depth

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Giving Transformer values their own residual-depth read, separate from the query/key read, improves validation negative log-likelihood in all 10 paired short-budget pretraining runs at 120M and 343M parameters.

  2. Do Value Vectors in Deep Layers Need Context from the Residual Stream?

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    Deep transformer layers can replace context-dependent value vectors with per-token lookup tables (Bank of Values), improving validation loss and the 21-benchmark average at 135M–780M while cutting FLOPs and the value cache.

  3. GaussianGPT: Towards Autoregressive 3D Gaussian Scene Generation

    cs.CV 2026-03 conditional novelty 6.0 of 10

    A causal transformer with 3D RoPE generates vector-quantized 3D Gaussian latent grids autoregressively, enabling unconditional synthesis, completion, and open-ended outpainting of indoor scenes.

  4. FEAT: Full-Dimensional Efficient Attention Transformer for Medical Video Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    FEAT combines spatial, temporal, and channel attention in a diffusion transformer and reports improved medical video generation metrics with lower parameter counts than Endora.

  5. Scaling Point-in-Time Language Models

    cs.CL 2026-04 conditional novelty 5.5 of 10

    Scaling point-in-time LLMs to 4B parameters and 1T temporally filtered tokens narrows the gap to unrestricted models to about 8–11 average points and yields positive out-of-sample Sharpe ratios from news embeddings.

Pith tools