Pith. sign in

REVIEW 12 cited by

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2103.10360 v2 pith:WEERGLUL submitted 2021-03-18 cs.CL cs.AIcs.LG

Zhengxiao Du , Yujie Qian , Xiao Liu , Ming Ding , Jiezhong Qiu , Zhilin Yang , Jie Tang This is my paper
classification cs.CLcs.AIcs.LG
keywords tasksbertmodelpretrainingautoregressiveblankgenerationlanguage
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

There have been various types of pretraining architectures including autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and encoder-decoder models (e.g., T5). However, none of the pretraining frameworks performs the best for all tasks of three main categories including natural language understanding (NLU), unconditional generation, and conditional generation. We propose a General Language Model (GLM) based on autoregressive blank infilling to address this challenge. GLM improves blank filling pretraining by adding 2D positional encodings and allowing an arbitrary order to predict spans, which results in performance gains over BERT and T5 on NLU tasks. Meanwhile, GLM can be pretrained for different types of tasks by varying the number and lengths of blanks. On a wide range of tasks across NLU, conditional and unconditional generation, GLM outperforms BERT, T5, and GPT given the same model sizes and data, and achieves the best performance from a single pretrained model with 1.25x parameters of BERT Large , demonstrating its generalizability to different downstream tasks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Constructing a Question-Answering Simulator through the Distillation of LLMs

    cs.LG 2025-09 conditional novelty 6.0 of 10

    LDSim distills an LLM's concept-prerequisite knowledge and mastery reasoning into a lightweight simulator that beats LLM-based and LLM-free baselines on four knowledge-tracing datasets.

  2. Beyond the Textual: Generating Coherent Visual Options for MCQs

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A four-stage framework (convertibility check, question/reason generation, optimal pair selection, and template-based image generation) produces MCQs with image options from ScienceQA content.

  3. CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 9,356-pair Chinese multimodal financial benchmark reveals that state-of-the-art multimodal LLMs, including GPT-4V, still score below 53% on objective and 39% on subjective financial chart tasks.

  4. EmotionTalk: An Interactive Chinese Multimodal Emotion Dataset With Rich Annotations

    cs.MM 2025-05 conditional novelty 6.0 of 10

    EmotionTalk provides 19,250 utterances from 744 Chinese dyadic dialogues with emotion, sentiment, and speaking-style caption annotations.

  5. AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training

    cs.LG 2025-05 conditional novelty 6.0 of 10

    AdamS replaces AdamW's second-moment storage with a momentum-and-gradient squared denominator, matching AdamW's loss curves with half the optimizer memory.

  6. Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation

    cs.CV 2026-07 conditional novelty 5.0 of 10

    A compact unified model that reuses a frozen VLM encoder and hybrid continuous/discrete tokens reaches competitive image understanding and generation with 15.6M training images and about $2,000 in compute.

  7. Pipelined Decoder for Efficient Context-Aware Text Generation

    cs.CL 2025-06 conditional novelty 5.0 of 10

    The pipelined decoder trains a T5 model to generate multiple subsequences in parallel with partial token dependencies, achieving 1.7x to 7x faster inference on context-aware generation tasks with roughly unchanged quality.

  8. Improving Factuality for Dialogue Response Generation via Graph-Based Knowledge Augmentation

    cs.CL 2025-06 conditional novelty 5.0 of 10

    The paper proposes TG-DRG and GA-DRG, two graph-augmented frameworks that combine coreference resolution, knowledge selection, and graph encoding to improve factuality of dialogue responses, evaluated with a newly pro...

  9. ALAS: An Automatic Latent Alignment Score for Audio Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    ALAS is a reference-based score for audio-text alignment in speech LLMs, computed from frozen hidden states and a Whisper-derived alignment path, with no training or fitted classifier.

  10. Large Language Model for Extracting Complex Contract Information in Industrial Scenes

    cs.CL 2025-07 conditional novelty 3.0 of 10

    Clustering contracts, LLM-based labeling, augmentation, and LoRA fine-tuning improve Chinese industrial contract field extraction over traditional TF-IDF/TextRank/SNOWNLP/KeyBERT baselines.

  11. Inference performance evaluation for LLMs on edge devices with a novel benchmarking framework and metric

    cs.PF 2025-08 reject novelty 2.0 of 10

    A benchmarking study of LLM inference on three edge devices with a proposed MBU metric that reduces to a standard throughput normalization.

  12. Eliciting Fine-Tuned Transformer Capabilities via Inference-Time Techniques

    cs.LG 2025-06 reject novelty 2.0 of 10

    The paper claims that in-context learning with finite example sets can approximate supervised fine-tuning in transformers, but the proof assumes the very approximation it sets out to establish.

Pith tools