REVIEW 18 cited by
BERT Rediscovers the Classical NLP Pipeline
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Pre-trained text encoders have rapidly advanced the state of the art on many NLP tasks. We focus on one such model, BERT, and aim to quantify where linguistic information is captured within the network. We find that the model represents the steps of the traditional NLP pipeline in an interpretable and localizable way, and that the regions responsible for each step appear in the expected sequence: POS tagging, parsing, NER, semantic roles, then coreference. Qualitative analysis reveals that the model can and often does adjust this pipeline dynamically, revising lower-level decisions on the basis of disambiguating information from higher-level representations.
Forward citations
Cited by 18 Pith papers
-
On the Interpretability of Whisper Encodings Using Sparse Autoencoders
Sparse autoencoders applied to Whisper ASR reveal monosemantic features across linguistic boundaries and demonstrate cross-lingual feature steering.
-
Epistemic Familiarity is Associated With Belief Stability in Large Language Models
Language models retract previously true answers far more often after seeing unfamiliar synthetic statements than after seeing familiar fictional statements, in both internal probes and prompted behavior.
-
Mobius Learning: Cyclic Depth Folding in Transformers
Möbius Learning, which cyclically shifts block order across data streams, achieves lower validation loss than fixed-order looped training at loop depths 6, 10, and 15 in a 124M-parameter GPT-2 experiment.
-
Prototype Transformer: Towards Language Model Architectures Interpretable by Design
ProtoT is an autoregressive language model whose attention is replaced by learned prototype channels that are claimed to capture nameable concepts and allow targeted edits, at linear sequence cost but with slightly lo...
-
Step-Wise Refusal Dynamics in Autoregressive and Diffusion Language Models
Diffusion language models can revise away harmful intermediate text, and a step-wise internal refusal signal detects jailbreaks cheaply across autoregressive and diffusion models.
-
Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning
Spectral features of attention are claimed to classify proof validity with near-perfect effect sizes, but the main evaluation relabels proofs using the classifier's own outputs.
-
LAYA: Layer-wise Attention Aggregation for Interpretable Depth-Aware Neural Networks
LAYA replaces the last-layer classifier with an input-conditioned attention mixture of all hidden representations, yielding small accuracy gains and per-sample layer-attribution scores.
-
AIM-CoT: Active Information-driven Multimodal Chain-of-Thought for Vision-Language Reasoning
AIM-CoT improves multimodal chain-of-thought by selecting image regions that reduce predictive uncertainty and inserting them when attention shifts toward the visual input.
-
All for One: LLMs Solve Mental Math at the Last Token With Information Transferred From Other Tokens
LLMs solve arithmetic in-context via an All-for-One pattern, with all input-specific computation occurring at the last token after a two-layer information transfer window.
-
STARE at the Structure: Steering ICL Exemplar Selection with Structural Alignment
A structure-aware exemplar retriever with a hidden-state syntactic injection module improves in-context semantic parsing across four benchmarks.
-
Meaning-infused grammar: Gradient Acceptability Shapes the Geometric Representations of Constructions in LLMs
In Pythia language models, the geometric distance between Double Object and Prepositional Object representations grows with human-rated preference strength, evidence for graded construction representations.
-
From Text to Actionable Intelligence: Automating STIX Entity and Relationship Extraction
Fine-tuned LLMs extract STIX entities and relationships from threat reports with per-module F1 scores of 84.4%, 88.5%, 95.5%, and 84.6%, backed by a new 4,011-entity annotated dataset.
-
Mechanistic Decomposition of Sentence Representations
Sentence embeddings can be decomposed into sparse, interpretable atoms via supervised dictionary learning, and mean pooling preserves mainly atoms aligned with the sentence direction.
-
Adaptive Task Vectors for Large Language Models
Adaptive Task Vectors use a small model to generate query-specific steering vectors for frozen LLMs, reporting strong accuracy and generalization, though the theoretical equivalences to LoRA and Prefix-Tuning are not ...
-
Similarity All The Way Up: Multilingual Generalization in LLMs Relies on Language-Level Similarity Structures
LLM embeddings largely reproduce the Indo-European language family tree, and how faithfully a model reproduces that tree correlates with its XNLI multilingual performance.
-
S2Sent: Nested Selectivity Aware Sentence Representation Learning
A lightweight cross-layer fusion module, S2Sent, improves unsupervised sentence embeddings by gating and DCT frequency selection across Transformer blocks.
-
SALMAN: Stability Analysis of Language Models Through the Maps Between Graph-based Manifolds
SALMAN ranks each text sample's fragility via the distortion between input and output embedding distances and uses the ranking to improve attack success rates and fine-tuning robustness.
-
PrunePEFT: Iterative Hybrid Pruning for Parameter-Efficient Fine-tuning of LLMs
An iterative hybrid pruning method selects which PEFT modules to keep at each transformer layer, matching or improving fixed PEFT baselines on GLUE at 1% trainable parameters.
Discussion (0). Sign in to comment.