Pith. sign in

REVIEW 23 cited by

Differential Transformer

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.05258 v2 pith:KVHGOYXX submitted 2024-10-07 cs.CL cs.LG

Tianzhu Ye , Li Dong , Yuqing Xia , Yutao Sun , Yi Zhu , Gao Huang , Furu Wei This is my paper · ORCID
classification cs.CLcs.LG
keywords transformerattentiondiffcontextdifferentialhallucinationin-contextirrelevant
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Transformer tends to overallocate attention to irrelevant context. In this work, we introduce Diff Transformer, which amplifies attention to the relevant context while canceling noise. Specifically, the differential attention mechanism calculates attention scores as the difference between two separate softmax attention maps. The subtraction cancels noise, promoting the emergence of sparse attention patterns. Experimental results on language modeling show that Diff Transformer outperforms Transformer in various settings of scaling up model size and training tokens. More intriguingly, it offers notable advantages in practical applications, such as long-context modeling, key information retrieval, hallucination mitigation, in-context learning, and reduction of activation outliers. By being less distracted by irrelevant context, Diff Transformer can mitigate hallucination in question answering and text summarization. For in-context learning, Diff Transformer not only enhances accuracy but is also more robust to order permutation, which was considered as a chronic robustness issue. The results position Diff Transformer as a highly effective and promising architecture to advance large language models.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 23 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers

    cs.LG 2025-07 conditional novelty 7.0 of 10

    Self-attention's local Lipschitz constant can be bounded using the attention probability distribution, and the softmax Jacobian spectral norm is shown to be at most 1/2, leading to a new robustness regularizer.

  2. Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KATA uses rank-one PSD feature maps to pack exponentially many nearly orthogonal keys at fixed interference, reaching near-softmax MQAR at 16× length with about a quarter of softmax's KV-cache entries.

  3. RTL-Sequencer: Towards Scalable RTL Timing Prediction with the Sequence-based Paradigm

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Linearizing RTL logic cones into breadth-first sequences and processing them with Mamba-2 sequence models yields better arrival-time, WNS, and TNS predictions than graph-based baselines on 21 open-source designs.

  4. Degradation-Aware Metric Prompting for Hyperspectral Image Restoration

    cs.CV 2025-12 conditional novelty 6.0 of 10

    A single hyperspectral-restoration model that computes six interpretable input statistics (frequency, texture, spectral-curvature metrics) and uses them as prompts to route Mixture-of-Experts modules achieves state-of...

  5. Multi-view Pyramid Transformer: Look Coarser to See Broader

    cs.CV 2025-12 conditional novelty 6.0 of 10

    MVP uses a two-level hierarchy of attention windows and token resolutions to reconstruct large 3D scenes from up to 256 input views in a single feed-forward pass, beating Long-LRM and iLRM on DL3DV and several zero-sh...

  6. Accelerated Machine Learning Force Field for Predicting Thermal Conductivity of Organic Liquids

    physics.chem-ph 2025-12 conditional novelty 6.0 of 10

    A single machine-learned force field predicts thermal conductivity of 20 organic liquids with ~14% MAPE by aligning simulated density to experiments.

  7. Differential Amplifier-Inspired AmpAttention for Multi-View Robotic Manipulation

    cs.RO 2026-07 conditional novelty 5.0 of 10

    AmpAttention and RVAF raise multi-view robotic manipulation success and cut training time by suppressing attention noise with a differential-amplifier-style mechanism plus a CMRR loss.

  8. CAT: Causal Attention Tuning For Injecting Fine-grained Causal Knowledge into Large Language Models

    cs.CL 2025-09 conditional novelty 5.0 of 10

    A training method that injects token-level causal labels into attention improves out-of-distribution accuracy on a synthetic benchmark and slightly on math/reasoning tasks.

  9. Novel Category Discovery with X-Agent Attention for Open-Vocabulary Semantic Segmentation

    cs.CV 2025-09 conditional novelty 5.0 of 10

    X-Agent adds agent tokens, chosen by optimal-transport affinity between text and visual keys, to CLIP attention, reporting marginal mIoU gains (0.1-0.6%) over prior OVSS methods.

  10. EIFNet: Leveraging Event-Image Fusion for Robust Semantic Segmentation

    cs.CV 2025-07 conditional novelty 5.0 of 10

    EIFNet fuses event and image features with three new attention modules and reports state-of-the-art segmentation accuracy on DDD17 and DSEC-Semantic.

  11. FD-DiT: Frequency Domain-Directed Diffusion Transformer for Low-Dose CT Reconstruction

    eess.IV 2025-06 conditional novelty 5.0 of 10

    A frequency-split diffusion transformer for low-dose CT denoising reports higher PSNR and SSIM than several recent methods on two benchmark datasets.

  12. Learning Bilateral Team Formation in Cooperative Multi-Agent Reinforcement Learning

    cs.MA 2025-06 conditional novelty 5.0 of 10

    A stable-matching-based team formation method improves generalization over a greedy score-based method in cooperative multi-agent RL.

  13. SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging

    cs.CV 2025-06 conditional novelty 5.0 of 10

    SEMA combines window attention with global token averaging, motivated by a dispersion theorem for generalized attention, and reports 0.2 to 0.7 percent top-1 accuracy gains over comparable vision Mamba and MILA models.

  14. ADFormer: Aggregation Differential Transformer for Passenger Demand Forecasting

    cs.AI 2025-06 conditional novelty 5.0 of 10

    ADFormer applies differential attention plus cluster-based spatial and temporal aggregation to passenger demand forecasting, reporting the best MAE and RMSE on most of nine test settings.

  15. Visible-Light Imaging Diagnosis of Neutral Particle Emission Tomography in the Tokamak Divertor: An Efficient Transformer-based Surrogate Model

    cs.CV 2026-07 conditional novelty 4.0 of 10

    A transformer-based network predicts 2D neutral-particle emission distributions in the EAST tokamak divertor from consecutive visible-light frames, achieving roughly 20 FPS with accuracy comparable to slower tomograph...

  16. DiffLoRA: Differential Low-Rank Adapters for Large Language Models

    cs.CL 2025-07 conditional novelty 4.0 of 10

    DiffLoRA, a low-rank adapter variant of differential attention, generally underperforms LoRA but shows gains on HumanEval and multi-value needle retrieval.

  17. Differential Attention for Multimodal Crisis Event Analysis

    cs.CV 2025-07 reject novelty 4.0 of 10

    On CrisisMMD, frozen CLIP embeddings with LLaVA-generated captions and Guided Cross Attention give the best accuracy, but the added Differential Attention layer does not improve, and sometimes reduces, performance.

  18. Differential Transformer-driven 6G Physical Layer for Collaborative Perception Enhancement

    eess.SP 2025-06 conditional novelty 4.0 of 10

    A Differential Transformer-based neural receiver outperforms a CNN-based receiver on simulated 6G V2X links and improves collaborative perception accuracy among four connected vehicles.

  19. Infant Cry Emotion Recognition Using Improved ECAPA-TDNN with Multiscale Feature Fusion and Attention Enhancement

    eess.AS 2025-06 conditional novelty 4.0 of 10

    An improved ECAPA-TDNN using multiscale channel attention and differential attention reports 82.20 percent accuracy on six-class infant cry emotion recognition on the iFLYTEK/USTC public dataset.

  20. CENet: Context Enhancement Network for Medical Image Segmentation

    cs.CV 2025-05 conditional novelty 4.0 of 10

    CENet uses boundary-enhanced skip connections and a multi-scale attention decoder to improve medical segmentation on four public benchmarks.

  21. ASDA: Audio Spectrogram Differential Attention Mechanism for Self-Supervised Representation Learning

    cs.SD 2025-07 conditional novelty 3.0 of 10

    Audio self-supervised learning with dual-softmax differential attention reports state-of-the-art numbers on AS-2M, AS20K, SPC-2, and ESC-50, but with tuning caveats and no code.

  22. Adaptive Differential Denoising for Respiratory Sounds Classification

    eess.AS 2025-06 conditional novelty 3.0 of 10

    An Adaptive Differential Denoising network achieves a 65.53% average score on ICBHI 2017 respiratory sound classification, surpassing the previous best by 1.99%.

  23. Differential Multimodal Transformers

    cs.AI 2025-07 reject novelty 2.0 of 10

    A proposed multimodal extension of differential attention turns out to be equivalent to a learned scalar multiplier on standard attention, so the reported retrieval gains do not support the claimed mechanism.

Pith tools