Pith. sign in

REVIEW 3 cited by

Adaptive Attention Span in Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1905.07799 v2 pith:76UZAUQC submitted 2019-05-19 cs.LG stat.ML

Sainbayar Sukhbaatar , Edouard Grave , Piotr Bojanowski , Armand Joulin This is my paper · ORCID
classification cs.LGstat.ML
keywords attentioncontextmaximumspanachieveadaptiveallowsapproach
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We propose a novel self-attention mechanism that can learn its optimal attention span. This allows us to extend significantly the maximum context size used in Transformer, while maintaining control over their memory footprint and computational time. We show the effectiveness of our approach on the task of character level language modeling, where we achieve state-of-the-art performances on text8 and enwiki8 by using a maximum context of 8k characters.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ALCo-FM: Adaptive Long-Context Foundation Model for Accident Prediction

    cs.LG 2025-07 conditional novelty 6.0 of 10

    ALCo-FM reports 0.92 F1 and 0.04 ECE for accident-risk prediction across 15 US cities, but the evaluation protocol is incompletely described.

  2. MoGU V2: Toward a Higher Pareto Frontier Between Model Usability and Security

    cs.CL 2025-09 conditional novelty 5.0 of 10

    MoGUv2 embeds small routers in the deeper layers of LLMs to dynamically blend a helpful variant and a refusal variant, improving safety against jailbreak and fine-tuning attacks while preserving usability.

  3. Change of Thought: Adaptive Test-Time Computation

    cs.LG 2025-07 reject novelty 4.0 of 10

    A transformer layer that iteratively refines its attention matrix to a fixed point is claimed to improve accuracy with no extra parameters, but the benchmark evidence is not reproducible.

Pith tools