REVIEW 15 cited by
MPNet: Masked and Permuted Pre-training for Language Understanding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
BERT adopts masked language modeling (MLM) for pre-training and is one of the most successful pre-training models. Since BERT neglects dependency among predicted tokens, XLNet introduces permuted language modeling (PLM) for pre-training to address this problem. However, XLNet does not leverage the full position information of a sentence and thus suffers from position discrepancy between pre-training and fine-tuning. In this paper, we propose MPNet, a novel pre-training method that inherits the advantages of BERT and XLNet and avoids their limitations. MPNet leverages the dependency among predicted tokens through permuted language modeling (vs. MLM in BERT), and takes auxiliary position information as input to make the model see a full sentence and thus reducing the position discrepancy (vs. PLM in XLNet). We pre-train MPNet on a large-scale dataset (over 160GB text corpora) and fine-tune on a variety of down-streaming tasks (GLUE, SQuAD, etc). Experimental results show that MPNet outperforms MLM and PLM by a large margin, and achieves better results on these tasks compared with previous state-of-the-art pre-trained methods (e.g., BERT, XLNet, RoBERTa) under the same model setting. The code and the pre-trained models are available at: https://github.com/microsoft/MPNet.
Forward citations
Cited by 15 Pith papers
-
RAG-Stack: Co-Optimizing RAG Serving Performance and Quality
RAG-Stack jointly optimizes RAG algorithm choices and serving-system settings via sub-metric-aware multi-objective Bayesian optimization plus an analytical performance model, reporting Pareto frontiers covering 52.5% ...
-
DragonCrawl: A Generative, Intent-Based Framework for Scalable Mobile End-to-End Testing
DragonCrawl uses GPT-4o-based intent reasoning, multimodal end-state detection, and backend tool calling to run 1,013 mobile regression tests in CI/CD, achieving over 91% pass rates and cutting test maintenance effort.
-
PARK: Personalized academic retrieval with knowledge-graphs
PARK personalizes academic search by embedding a citation-derived knowledge graph into the same vector space as a neural retrieval model, beating baselines in three of four domains.
-
An Automated Length-Aware Quality Metric for Summarization
NOIR is a reference-free summarization metric that divides the logarithm of token compression by the logarithm of embedding-based semantic retention.
-
Intertextual Parallel Detection in Biblical Hebrew: A Transformer-Based Benchmark
E5 and AlephBERT embeddings rank known Samuel/Kings-Chronicles parallels above non-parallel verses, but the benchmark restricts searches to Samuel/Kings and lacks baselines.
-
Revealing the Numeracy Gap: An Empirical Investigation of Text Embedding Models
Across 13 embedding models and 18 numeric formats, retrieval accuracy on the new EmbedNum-1K benchmark averages 54%, just above chance, showing that embedding models largely fail to encode numeric detail.
-
An Agentic Model Context Protocol Framework for Medical Concept Standardization
An MCP-based LLM agent with mandatory Athena lookups achieved 100% retrieval success on 150 OMOP terms and scored higher on clinical relevance than historical human mappings.
-
InSQuAD: In-Context Learning for Efficient Retrieval via Submodular Mutual Information to Enforce Quality and Diversity
A submodular mutual information framework for selecting and training in-context learning exemplars improves average accuracy on nine benchmarks by about five points over the IDEAL baseline.
-
Verified Language Processing with Hybrid Explainability: A Technical Report
A logic-based NLP pipeline converts sentences to first-order logic and uses possible-world truth tables to classify implication, inconsistency, and indifference, with perfect scores on three small self-built datasets.
-
Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation
A systematic evaluation shows that recursive 100-token non-overlapping chunks and retrieval-tuned embeddings outperform fixed-size chunks and domain-specific models like SciBERT for chemistry retrieval, and it introdu...
-
Quantifying Misattribution Unfairness in Authorship Attribution
Authorship attribution models misattribute texts to some authors far more often than chance, and the risk is highest for authors whose author embeddings sit near the centroid.
-
Improving Dialogue State Tracking through Combinatorial Search for In-Context Examples
A combinatorial scoring method for retriever training data improves few-shot dialogue state tracking by 20x in data efficiency and by 12% in oracle upper-bound JGA over prior methods.
-
How Small Transformation Expose the Weakness of Semantic Similarity Measures
A diagnostic benchmark of text and code transformations finds embedding similarity metrics often conflate opposition with equivalence; LLM judges discriminate better, and Euclidean distance improves code embeddings.
-
Detoxify: A framework for abusive text transformation using LLMs
A comparative study claims Groq produces the most positive but least semantically faithful detoxified text, but the comparison is undermined by inconsistent methodology.
-
Surfacing Semantic Orthogonality Across Model Safety Benchmarks: A Multi-Dimensional Analysis
Embedding-based clustering of five safety benchmarks reveals six rough harm themes, with datasets showing uneven topic coverage such as GretelAI on privacy and WildGuardMix on self-harm.
Discussion (0). Sign in to comment.