arxiv: 2604.05158 · v2 · submitted 2026-04-06 · 💻 cs.CL

Recognition: no theorem link

Just Pass Twice: Efficient Token Classification with LLMs for Zero-Shot NER

Ahmed Ewais , Ahmed Hashish , Amr Ali

Authors on Pith no claims yet

Pith reviewed 2026-05-10 19:10 UTC · model grok-4.3

classification 💻 cs.CL

keywords zero-shot named entity recognitionlarge language modelscausal attentiontoken classificationefficient inferencedefinition-guided embeddings

0 comments

The pith

Concatenating the input sentence to itself gives causal LLMs full context for zero-shot named entity recognition.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that a simple duplication of the input sentence lets causal large language models perform token classification with access to the full sentence. This matters because named entity recognition often requires looking at words both before and after a candidate entity to decide its type. Existing ways to use LLMs for this either ignore the future context or generate text slowly and unreliably. The duplication trick combined with entity definitions from text turns the model into an effective zero-shot classifier that works across different domains.

Core claim

The central discovery is that concatenating an input sentence to itself within a single forward pass allows each token in the second instance to attend to the complete original sentence through causal attention. This provides bidirectional context for token-level classification without modifying the model architecture. When paired with entity type embeddings constructed from natural language definitions, the resulting representations support accurate zero-shot generalization to unseen domains and entity types.

What carries the argument

The Just Pass Twice concatenation, which duplicates the input sequence so that the second copy receives full preceding context from the first copy.

If this is right

Delivers state-of-the-art performance on zero-shot NER tasks across multiple benchmarks.
Outperforms prior methods by an average of 7.9 F1 points on CrossNER and MIT datasets.
Runs over 20 times faster than generative LLM approaches for the same task.
Reduces issues like hallucinated entities and output formatting errors common in generative methods.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

This approach could extend to other sequence labeling tasks such as part-of-speech tagging where future context aids disambiguation.
Combining it with lightweight fine-tuning on definitions might further improve domain adaptation without full retraining.
Testing on very long documents would reveal if the doubled length causes attention dilution or context window issues.

Load-bearing premise

That simply concatenating the input sentence to itself lets tokens in the second pass reliably use future context for accurate classification without introducing artifacts, and that definition-guided entity embeddings suffice for flexible zero-shot generalization across domains.

What would settle it

Testing the method on a collection of sentences where correct entity labels depend on words appearing after the entity, and finding no accuracy gain over single-pass baselines or inconsistent labels between the two copies, would show the claim does not hold.

Figures

Figures reproduced from arXiv: 2604.05158 by Ahmed Ewais, Ahmed Hashish, Amr Ali.

**Figure 1.** Figure 1: Just Pass Twice (JPT) enables bidirectional token classification in causal LLMs. (Top) Standard causal masking restricts the target token “Paris” from attending to future context like “album” (red barrier), leading to ambiguity. (Bottom) JPT duplicates the input sequence. In the second pass, the target token (green box) attends backwards to the entire original sequence (green arrows), resolving the entity … view at source ↗

**Figure 2.** Figure 2: Architecture of the proposed model. (Top) Entity definitions are encoded via a text encoder (dimension denc) and projected into the shared space R dp using the Entity Projection MLP, yielding entity embeddings pper, ploc, . . . (Bottom) The Causal LLM processes the duplicated input sequence. Hidden states from the second pass (h5, . . . , h7) are projected to token embeddings t1, . . . , tn ∈ R dp and sco… view at source ↗

**Figure 3.** Figure 3: Attention weights averaged across all trans [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

**Figure 4.** Figure 4: Distribution of entity type frequencies in [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗

**Figure 5.** Figure 5: Training examples showing fine-grained entity types across diverse domains. Entity spans are [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗

**Figure 6.** Figure 6: Complete prompt template for JPT. The system prompt specifies output format and labeling rules. Entity definitions are injected in the first user turn, and the input text is duplicated with explicit markers in the second user turn. Model AI Lit. Music Politics Science Avg. Qwen3-4B 49.9 47.9 74.7 56.2 58.1 57.4 JPT-4B 68.3 73.7 84.1 76.4 69.5 74.4 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗

**Figure 7.** Figure 7: Additional attention visualizations. Rows: second-pass tokens. Columns: first-pass tokens. The diagonal [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗

**Figure 8.** Figure 8: Entity type confusion matrix aggregated across CrossNER and MIT. Most confusions occur between [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗

**Figure 9.** Figure 9: Examples from CrossNER Politics where our model correctly identifies fine-grained entity types. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗

read the original abstract

Large language models encode extensive world knowledge valuable for zero-shot named entity recognition. However, their causal attention mechanism, where tokens attend only to preceding context, prevents effective token classification when disambiguation requires future context. Existing approaches use LLMs generatively, prompting them to list entities or produce structured outputs, but suffer from slow autoregressive decoding, hallucinated entities, and formatting errors. We propose Just Pass Twice (JPT), a simple yet effective method that enables causal LLMs to perform discriminative token classification with full bidirectional context. Our key insight is that concatenating the input to itself lets each token in the second pass attend to the complete sentence, requiring no architectural modifications. We combine these representations with definition-guided entity embeddings for flexible zero-shot generalization. Our approach achieves state-of-the-art results on zero-shot NER benchmarks, surpassing the previous best method by +7.9 F1 on average across CrossNER and MIT benchmarks, being over 20x faster than comparable generative methods.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Duplicating the input sentence gives causal LLMs a workable but imperfect route to more context for zero-shot NER token classification, with solid efficiency claims but a real question on whether the context is truly bidirectional.

read the letter

The main contribution is the Just Pass Twice trick: feed the sentence to itself so the second copy's tokens can attend across the full first copy. Pair that with definition-guided embeddings and you get zero-shot token classification without generative prompting or model changes. The reported outcome is +7.9 F1 over prior best on CrossNER and MIT, plus a 20x speed-up over generative baselines. That combination of simplicity and claimed performance is the part worth noticing first.

Referee Report

1 major / 3 minor

Summary. The paper introduces Just Pass Twice (JPT), a method for zero-shot NER with causal LLMs. By concatenating each input sentence to itself, tokens in the second pass can attend to the full original sentence as context without architectural changes. These representations are combined with definition-guided entity embeddings for flexible zero-shot generalization. The approach is reported to achieve SOTA results, surpassing the prior best method by +7.9 F1 on average across CrossNER and MIT benchmarks while being over 20x faster than generative LLM baselines.

Significance. If the empirical gains hold under rigorous verification, JPT offers a lightweight way to repurpose causal LLMs for discriminative token-level tasks, avoiding the latency and hallucination issues of generative decoding. The reported speed-up and simplicity are practical strengths; the paper supplies benchmark results on standard zero-shot NER suites.

major comments (1)

[§3] §3 (Method, S+S construction): The central claim that second-pass tokens obtain 'full bidirectional context' is load-bearing for the novelty and the +7.9 F1 result. Because attention remains strictly causal, hidden states for positions 1..L in the first copy contain only left-of-position information; a token at L+k attending to position j < L therefore receives a key/value that has never seen tokens after j. The manuscript must either (a) derive why this partial prefix still supplies the future-context disambiguation needed for NER or (b) supply an ablation (e.g., comparing against a true bidirectional encoder or against a non-duplicated causal baseline) that isolates the contribution of the duplicated prefix. Without this, the mechanism's sufficiency for the reported gains remains unverified.

minor comments (3)

[Table 1, §4.2] Table 1 and §4.2: report per-benchmark F1 scores, standard deviations across runs, and the exact identity of the 'previous best method' that is surpassed by +7.9. Include statistical significance tests.
[§4.3] §4.3 (Ablations): add a control that removes the definition-guided embeddings to quantify their isolated contribution versus the S+S construction alone.
[Figure 2] Figure 2 (attention visualization): clarify whether the plotted attention heads are from the first or second pass and whether position embeddings are shared or reset at the concatenation point.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their thorough review and for identifying this key subtlety in the S+S construction. We address the comment below and commit to revisions that clarify the mechanism and strengthen the empirical support.

read point-by-point responses

Referee: §3 (Method, S+S construction): The central claim that second-pass tokens obtain 'full bidirectional context' is load-bearing for the novelty and the +7.9 F1 result. Because attention remains strictly causal, hidden states for positions 1..L in the first copy contain only left-of-position information; a token at L+k attending to position j < L therefore receives a key/value that has never seen tokens after j. The manuscript must either (a) derive why this partial prefix still supplies the future-context disambiguation needed for NER or (b) supply an ablation (e.g., comparing against a true bidirectional encoder or against a non-duplicated causal baseline) that isolates the contribution of the duplicated prefix. Without this, the mechanism's sufficiency for the reported gains remains unverified.

Authors: We appreciate the referee's precise analysis of the causal attention constraints. We agree that the first-pass hidden states encode only left-of-position information, so second-pass tokens do not receive true future context from the first copy; the method does not achieve full bidirectionality equivalent to a bidirectional encoder. Instead, duplication supplies each second-pass token with the full sentence as prior context (causally encoded) plus its own left context within the second pass. This enables discriminative classification on unmodified causal LLMs. While we lack a formal derivation proving sufficiency for every NER disambiguation case, the observed +7.9 F1 gains over single-pass and generative baselines indicate practical effectiveness. In the revision we will (a) update §3 to describe the context as 'enhanced sentence-level context via duplication under causal constraints' rather than 'full bidirectional context' and (b) add an ablation comparing JPT to a non-duplicated causal baseline to isolate the duplicated prefix's contribution. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical method without load-bearing derivations

full rationale

The paper presents an empirical technique (input concatenation for second-pass token classification) whose central claims are validated by benchmark results rather than any closed mathematical derivation. No equations, fitted parameters, or self-citation chains are used to derive the performance claims; the method is described as a simple insight requiring no architectural changes, and results are reported as experimental outcomes on CrossNER and MIT datasets. This is self-contained against external benchmarks with no reduction of predictions to inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The method rests on the assumption that standard causal attention on a duplicated input sequence behaves as intended for token-level decisions and that entity definitions can be turned into useful guiding embeddings without further training.

axioms (2)

domain assumption Causal LLMs can use a duplicated input sequence to give each token access to the full original sentence without introducing harmful artifacts.
This is the central mechanism that converts unidirectional attention into effective bidirectional context.
domain assumption Definition-guided entity embeddings enable reliable zero-shot generalization across different entity type sets.
The paper relies on this to avoid task-specific fine-tuning.

pith-pipeline@v0.9.0 · 5469 in / 1353 out tokens · 40449 ms · 2026-05-10T19:10:10.910661+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

8 extracted references · 2 canonical work pages · 1 internal anchor

[1]

DeBERTa: Decoding-enhanced BERT with Disentangled Attention

NuNER: Entity recognition encoder pre- training via LLM-annotated data. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11829–11841, Miami, Florida, USA. Association for Computational Linguistics. Zifeng Cheng, Zhaoling Chen, Zhiwei Jiang, Yafeng Yin, Cong Wang, Shiping Ge, and Qing Gu. 2025. Multi-prompting...

work page internal anchor Pith review arXiv 2024
[2]

Preprint, arXiv:2401.10825

Recent advances in named entity recogni- tion: A comprehensive survey and comparative study. Preprint, arXiv:2401.10825. Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. 2023. Evaluating chatgpt’s information extraction capabilities: An as- sessment of performance, explainability, calibration, and faithfulness.Preprint, ar...

work page arXiv 2023
[3]

agent,” “place

TELEClass: Taxonomy enrichment and llm- enhanced hierarchical text classification with mini- mal supervision. InWWW. GitHub repository. Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. 2024. UniversalNER: Targeted distil- lation from large language models for open named entity recognition. InThe Twelfth International Con- ference on Learnin...

2024
[4]

Athlete,

Type Generation: Given a passage and its topic hierarchy, the model proposes domain- appropriate entity types with definitions (e.g., “Athlete,” “Team,” “Stadium” for sports arti- cles). Statistic Value Total sentences 17,489 Total tokens 3,391,899 Total entity mentions 374,705 Unique entity types 5,009 Avg. sentence length 194.9 tokens Entity/non-entity ...
[5]

For quality validation, Claude Opus 4.5 assesses random samples across entity type appropriateness, definition actionability, and extraction accuracy

Entity Detection: The model identifies entity spans in the text, followed by a gap-detection pass to catch missed mentions. For quality validation, Claude Opus 4.5 assesses random samples across entity type appropriateness, definition actionability, and extraction accuracy. The resulting dataset comprises17,489training examples with5,009entity types and2,...
[6]

Specify inclusions and exclusions: What counts and what doesn’t
[7]

Provide concrete examples: Representative instances of the type
[8]

A geographical place

Address ambiguities: Clarify edge cases (e.g., does “nearby” count as location?) E Additional Ablations This section presents additional ablation stud- ies that analyze the effect of model scale and parameter-efficient adaptation onJPTperformance. These experiments help characterize the trade-offs between accuracy, model capacity, and inference efficiency...

2023