Pith. sign in

REVIEW 5 cited by

Contextual Position Encoding: Learning to Count What's Important

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.18719 v2 pith:JJPT672S submitted 2024-05-29 cs.CL cs.AI

Olga Golovneva , Tianlu Wang , Jason Weston , Sainbayar Sukhbaatar This is my paper · ORCID
classification cs.CLcs.AI
keywords positionencodingallowsattendingcontextualcopei-thlanguage
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

The attention mechanism is a critical component of Large Language Models (LLMs) that allows tokens in a sequence to interact with each other, but is order-invariant. Incorporating position encoding (PE) makes it possible to address by position, such as attending to the i-th token. However, current PE methods use token counts to derive position, and thus cannot generalize to higher levels of abstraction, such as attending to the i-th sentence. In this paper, we propose a new position encoding method, Contextual Position Encoding (CoPE), that allows positions to be conditioned on context by incrementing position only on certain tokens determined by the model. This allows more general position addressing such as attending to the $i$-th particular word, noun, or sentence. We show that CoPE can solve the selective copy, counting and Flip-Flop tasks where popular position embeddings fail, and improves perplexity on language modeling and coding tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Addressable Memory for Video World Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Video world models can recall revisited scenes far beyond their training horizon by storing compressed memory at fixed in-distribution positions and averaging keys in a rotation-free space.

  2. Sample Complexity and Representation Ability of Test-time Scaling Paradigms

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Best-of-n sampling provably needs about 1/Δ samples versus 1/Δ² for self-consistency, and a constructed Transformer can route among experts using verifier feedback to reach near-optimal final responses.

  3. A Contextual-Aware Position Encoding for Sequential Recommendation

    cs.IR 2025-02 conditional novelty 5.0 of 10

    CAPE computes context-dependent positions from item dissimilarity and fuses them into attention, improving sequential recommenders on benchmark and live-traffic tests.

  4. HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models

    cs.CL 2025-09 reject novelty 4.0 of 10

    HoPE replaces RoPE's sine/cosine rotations with hyperbolic functions plus an exponential damping term to enforce monotonic attention decay, but the claimed consistent superiority and the 'RoPE as special case' theorem...

  5. Latent Multi-Head Attention for Small Language Models

    cs.CL 2025-06 conditional novelty 4.0 of 10

    MLA with rotary embeddings at half latent rank keeps validation loss nearly unchanged while cutting KV-cache memory by roughly half on small language models.

Pith tools