A canary injection protocol for linking observed AI agent behavior to the responsible account at the hosting vendor, with robust variants for adversarial filtering.
hub
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
42 Pith papers cite this work, alongside 25 external citations. Polarity classification is still indexing.
42
Pith papers citing it
25
external citations · Pith
abstract
Encoder-only transformer models such as BERT offer a great performance-size tradeoff for retrieval and classification tasks with respect to larger decoder-only models. Despite being the workhorse of numerous production pipelines, there have been limited Pareto improvements to BERT since its release. In this paper, we introduce ModernBERT, bringing modern model optimizations to encoder-only models and representing a major Pareto improvement over older encoders. Trained on 2 trillion tokens with a native 8192 sequence length, ModernBERT models exhibit state-of-the-art results on a large pool of evaluations encompassing diverse classification tasks and both single and multi-vector retrieval on different domains (including code). In addition to strong downstream performance, ModernBERT is also the most speed and memory efficient encoder and is designed for inference on common GPUs.
hub tools
citation-role summary
background 1
method 1
citation-polarity summary
representative citing papers
MaxSim similarity can exactly replicate inner products of non-negative sparse vectors of arbitrary dimension, and a proposed Signed MaxSim extension enables exact replication for arbitrary real-valued vectors.
DT² trains digital twins to preserve pairwise policy rankings from fitted Q-evaluation on offline data rather than minimizing one-step transition errors, improving policy ranking and reducing decision regret.
LEDGER provides a corpus of 4,999 annual reports with 31 labeled KPIs and three benchmarks for page-level retrieval, needle-in-haystack lookup, and full KPI extraction from long documents.
A Dutch BERT model encodes gender linearly by epoch 20 but does not dynamically update its representations when explicit female cues contradict learned stereotypical associations in short sentence templates.
Masked-position MLM plus JEPA latent prediction outperforms MLM-only pretraining on 10-11 of 16 downstream tasks for 35M-150M protein models while JEPA alone fails.
HyperTransport amortizes activation steering for T2I models via a hypernetwork that predicts intervention parameters from CLIP embeddings, delivering 3600-7000x speedup and matching per-concept baselines on 167 unseen concepts.
Monitors trained on prompt-elicited reward-hacking trajectories fail to generalize to hacking behaviors that arise naturally during RL training of code models, whereas trajectories curated by Trace-and-Amplify transfer better and to unseen hack types.
Dual Triangle Attention achieves effective bidirectional attention with built-in positional inductive bias via dual triangular masks, outperforming standard bidirectional attention on position-sensitive tasks and showing strong masked language modeling results with or without positional embeddings.
Retrocausal transformer decomposes multi-agent motion forecasts into marginals and pairwise joints, models uncertainty with compressed exponentials, achieves strong Waymo results, generalizes to Argoverse 2 and V2X-Seq, and enables implicit instruction following from standard training.
Fine-tuned ModernBERT-family encoders match LLM judges on F1, false negative rate, and precision-recall for harmful output detection across adversarial datasets and attack types while promising lower cost and latency.
A local cascade framework for educational dialogue de-identification reaches 0.958 macro F1 on math tutoring transcripts, outperforming same-family LLM-only and commercial baselines while remaining fully on-device.
AbstRAG introduces reflective refinement to close abstraction gaps in retrieval-augmented generation by decomposing mismatches into expression, conceptual, intent-evidence, and event-type components and scoring with match quality, utility prior, and bridge cost.
LLM routers across 21 methods on 5 benchmarks converge to similar accuracy below oracle due to learning global performance trends rather than fine-grained query signals.
SAVER proposes a conformal groundability gate plus submodular image selector that activates vision only when needed for multimodal named entity recognition and relation extraction, improving F1 while lowering compute.
HyDRA routes queries to cost-effective LLMs by predicting multi-dimensional capability requirements with a multi-head encoder and applying shortfall matching against configuration-defined model profiles, delivering up to 72.5 percent cost savings on coding benchmarks while remaining decoupled from具体
GLiGuard is a compact schema-conditioned bidirectional encoder that matches 7B-27B guard models on safety benchmarks while delivering up to 16x higher throughput and 17x lower latency.
NorBERTo, a ModernBERT-style Portuguese encoder trained from scratch on the 331B-token Aurora-PT corpus, posts top scores on PLUE and ASSIN 2 entailment, but lower scores on semantic similarity.
A novel supervised predictor modeling semantic relationships among question, retrieved passages, and generated answer best forecasts when RAG improves QA performance.
Explanation biases in feature attribution methods are systematic products of lexical and positional preferences, with observed trade-offs across models and higher bias in anomalous explanations.
PMSR progressively constructs structured reasoning trajectories with dual-scope queries and compositional reasoning to improve knowledge acquisition and answer accuracy in knowledge-intensive VQA.
Treatment policies from multimodal EHRs improve when doubly robust pseudo-outcomes are built from annotated confounders and then regressed onto text-and-tabular representations, rather than estimating effects directly from those representations.
Controlled ablations of 38 models find MLM superior to CLM on representation benchmarks while CLM offers better data efficiency and stability; a biphasic CLM-then-MLM schedule is optimal under fixed compute and improves when initialized from pretrained CLM models.
Architectures with stronger inductive biases attain higher in-distribution accuracy but degrade faster under temporal distribution shift, while frozen pretrained encoders trade accuracy for stability.
citing papers explorer
-
Who Owns This Agent? Tracing AI Agents Back to Their Owners
A canary injection protocol for linking observed AI agent behavior to the responsible account at the hosting vendor, with robust variants for adversarial filtering.
-
Quantifying and Expanding the Theoretical Capacity of Late-Interaction Retrieval Models
MaxSim similarity can exactly replicate inner products of non-negative sparse vectors of arbitrary dimension, and a proposed Signed MaxSim extension enables exact replication for arbitrary real-valued vectors.
-
$\text{DT}^2$: Decision-Targeted Digital Twins
DT² trains digital twins to preserve pairwise policy rankings from fitted Q-evaluation on offline data rather than minimizing one-step transition errors, improving policy ranking and reducing decision regret.
-
LEDGER: A Long-Context Benchmark of Corporate Annual Reports for Grounded Financial Retrieval and Extraction
LEDGER provides a corpus of 4,999 annual reports with 31 labeled KPIs and three benchmarks for page-level retrieval, needle-in-haystack lookup, and full KPI extraction from long documents.
-
Is She Even Relevant? When BERT Ignores Explicit Gender Cues
A Dutch BERT model encodes gender linearly by epoch 20 but does not dynamically update its representations when explicit female cues contradict learned stereotypical associations in short sentence templates.
-
ProteinJEPA: Latent prediction complements protein language models
Masked-position MLM plus JEPA latent prediction outperforms MLM-only pretraining on 10-11 of 16 downstream tasks for 35M-150M protein models while JEPA alone fails.
-
HyperTransport: Amortized Conditioning of T2I Generative Models
HyperTransport amortizes activation steering for T2I models via a hypernetwork that predicts intervention parameters from CLIP embeddings, delivering 3600-7000x speedup and matching per-concept baselines on 167 unseen concepts.
-
Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Training-Time Reward Hacking in Code Generation
Monitors trained on prompt-elicited reward-hacking trajectories fail to generalize to hacking behaviors that arise naturally during RL training of code models, whereas trajectories curated by Trace-and-Amplify transfer better and to unseen hack types.
-
Dual Triangle Attention: Effective Bidirectional Attention Without Positional Embeddings
Dual Triangle Attention achieves effective bidirectional attention with built-in positional inductive bias via dual triangular masks, outperforming standard bidirectional attention on position-sensitive tasks and showing strong masked language modeling results with or without positional embeddings.
-
RetroMotion: Retrocausal Motion Forecasting Models are Instructable
Retrocausal transformer decomposes multi-agent motion forecasts into marginals and pairwise joints, models uncertainty with compressed exponentials, achieves strong Waymo results, generalizes to Argoverse 2 and V2X-Seq, and enables implicit instruction following from standard training.
-
Do Encoders Suffice? A Systematic Comparison of Encoder and Decoder Safety Judges for LLM Adversarial Evaluation
Fine-tuned ModernBERT-family encoders match LLM judges on F1, false negative rate, and precision-recall for harmful output detection across adversarial datasets and attack types while promising lower cost and latency.
-
Redact or Keep? A Fully Local AI Cascade for Educational Dialogue De-Identification
A local cascade framework for educational dialogue de-identification reaches 0.958 macro F1 on math tutoring transcripts, outperforming same-family LLM-only and commercial baselines while remaining fully on-device.
-
AbstRAG: Learning to Abstract for Retrieval Problems
AbstRAG introduces reflective refinement to close abstraction gaps in retrieval-augmented generation by decomposing mismatches into expression, conceptual, intent-evidence, and event-type components and scoring with match quality, utility prior, and bridge cost.
-
The Routing Plateau: Understanding and Breaking the Accuracy Limits of LLM Routers
LLM routers across 21 methods on 5 benchmarks converge to similar accuracy below oracle due to learning global performance trends rather than fine-grained query signals.
-
SAVER: Selective As-Needed Vision Evidence for Multimodal Information Extraction
SAVER proposes a conformal groundability gate plus submodular image selector that activates vision only when needed for multimodal named entity recognition and relation extraction, improving F1 while lowering compute.
-
HyDRA: Hybrid Dynamic Routing Architecture for Heterogeneous LLM Pools
HyDRA routes queries to cost-effective LLMs by predicting multi-dimensional capability requirements with a multi-head encoder and applying shortfall matching against configuration-defined model profiles, delivering up to 72.5 percent cost savings on coding benchmarks while remaining decoupled from具体
-
GLiGuard: Schema-Conditioned Classification for LLM Safeguard
GLiGuard is a compact schema-conditioned bidirectional encoder that matches 7B-27B guard models on safety benchmarks while delivering up to 16x higher throughput and 17x lower latency.
-
NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpus
NorBERTo, a ModernBERT-style Portuguese encoder trained from scratch on the 331B-token Aurora-PT corpus, posts top scores on PLUE and ASSIN 2 entailment, but lower scores on semantic similarity.
-
Rag Performance Prediction for Question Answering
A novel supervised predictor modeling semantic relationships among question, retrieved passages, and generated answer best forecasts when RAG improves QA performance.
-
Explanation Bias is a Product: Revealing the Hidden Lexical and Position Preferences in Post-Hoc Feature Attribution
Explanation biases in feature attribution methods are systematic products of lexical and positional preferences, with observed trade-offs across models and higher bias in anomalous explanations.
-
Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering
PMSR progressively constructs structured reasoning trajectories with dual-scope queries and compositional reasoning to improve knowledge acquisition and answer accuracy in knowledge-intensive VQA.
-
Annotation-Assisted Learning of Treatment Policies From Multimodal Electronic Health Records
Treatment policies from multimodal EHRs improve when doubly robust pseudo-outcomes are built from annotated confounders and then regressed onto text-and-tabular representations, rather than estimating effects directly from those representations.
-
Should We Still Pretrain Encoders with Masked Language Modeling?
Controlled ablations of 38 models find MLM superior to CLM on representation benchmarks while CLM offers better data efficiency and stability; a biphasic CLM-then-MLM schedule is optimal under fixed compute and improves when initialized from pretrained CLM models.
-
Drift Happens: An Empirical Study of Neural Architecture Robustness to Temporal Distribution Shift
Architectures with stronger inductive biases attain higher in-distribution accuracy but degrade faster under temporal distribution shift, while frozen pretrained encoders trade accuracy for stability.
-
Do Safety Guardrails Need to Reason? LeanGuard: A Fast and Light Approach for Robust Moderation
A 395M label-only bidirectional encoder achieves 82.90 average F1 on moderation benchmarks without reasoning, matching larger reasoning decoders at ~100x lower inference cost and with better robustness properties.
-
Unified Multi-Task Relevance Modeling for E-Commerce: Comparing Task Routing Architectures Across LLMs and Cross-Encoders
A multi-head private-layer ensemble in a unified multi-task setup reaches 89.96% accuracy on 453K e-commerce examples and improves low-resource tasks by up to 14% while revealing encoder-decoder asymmetry in task identity handling.
-
ConcernBERT: Learning Responsibilities Using Class Membership
ConcernBERT is a BERT embedding model trained with triplet loss on class membership to encode concern-level semantics in Java entities, evaluated by recovering original classes from merged unlabeled groups on a new dataset of over 2M files, outperforming existing models.
-
Efficient Financial Language Understanding via Distillation with Synthetic Data
A clustering-based synthetic data distillation framework enables compact models to match or exceed a large teacher on financial sentiment analysis using only a small set of real labeled examples.
-
Crossing the Validation Crisis: Cross-Validation Reduces Benchmarking Variance Surprisingly Well
Multiple cross-validation splits reduce variance in ML benchmarking estimates via a new sample-gain metric, shown on synthetic data plus histopathology and NLP tasks.
-
When Does Synthetic Patent Data Help? Volume-Fidelity Trade-offs in Low-Resource Multi-Label Classification
Synthetic data boosts multi-label patent classification mainly through volume in low-data regimes, with fidelity mattering more as real data increases and a 20-30% real data mix optimal under fixed budgets.
-
Response-free item difficulty modelling for multiple-choice items with fine-tuned transformers: Component-wise representation and multi-task learning
Fine-tuned transformers with multi-task learning recover substantial wording-derived signal for item difficulty at small sample sizes typical in applied testing.
-
Efficient Listwise Reranking with Compressed Document Representations
RRK compresses documents to multi-token embeddings for efficient listwise reranking, enabling an 8B model to achieve 3x-18x speedups over smaller models with comparable or better effectiveness.
-
Commonsense Knowledge with Negation: A Resource to Enhance Negation Understanding
Augmenting commonsense knowledge corpora with negation produces over 2M new triples that benefit LLM negation understanding when used for pre-training.
-
Brick: Spatial Capability Routing for the Mixture-of-Models (MoM) Paradigm
Brick routes queries to LLMs using capability scores and difficulty estimates, reaching 76.98% accuracy at max-quality and 4.71x lower cost at neutral profile on 5,504 queries versus always using the strongest model.
-
KliniskVestBERT: BERT Model Specialised to Norwegian Clinical Texts
Three BERT models are further pre-trained on Norwegian clinical notes and discharge summaries, then shown to outperform their base models on synthetic clinical benchmarks and real-world tasks.
-
DunbaaBERT: From Sacrifice to Semantics
DunbaaBERT releases competitive Urdu encoder models trained from scratch, with the 32k-vocab variant showing the best efficiency profile across acceptability, classification, and sentiment tasks.
-
m3BERT: A Modern, Multi-lingual, Matryoshka Bidirectional Encoder
m3BERT uses a three-stage Matryoshka pretraining approach on a bidirectional encoder to support variable embedding sizes while outperforming prior models on large-scale retrieval tasks.
-
Filter-then-Verify: A Multiphase GNN and ModernBERT Framework for Social Engineering Detection in Email Networks
A two-stage GNN-plus-ModernBERT framework detects social engineering attacks in email networks by first filtering structural anomalies at 86% recall and then verifying content to reach over 92% precision on augmented Enron data.
-
Depression Detection at the Point of Care: Automated Analysis of Linguistic Signals from Routine Primary Care Encounters
Zero-shot GPT-OSS detects depression from 1,108 primary care encounter transcripts with AUPRC 0.51 and AUROC 0.77, with meaningful signals in the first 128 patient tokens and added value from dyadic mirroring.
-
A Unified Framework for Modeling Heterogeneous Financial Data via Dual-Granularity Prompting
FinLangNet applies dual-granularity prompting in a sequential model to heterogeneous financial data, reporting 6.3 pp KS improvement and 9.9% bad debt reduction in real-world deployment.
-
Legal Domain Adaptation of Modern BERT Models
Further pre-training ModernBERT on US court opinions improves results on legal datasets compared to the base model, with gains similar to early BERT domain adaptation work.
- Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction