REVIEW 7 cited by
PanGu-{\Sigma}: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The scaling of large language models has greatly improved natural language understanding, generation, and reasoning. In this work, we develop a system that trained a trillion-parameter language model on a cluster of Ascend 910 AI processors and MindSpore framework, and present the language model with 1.085T parameters named PanGu-{\Sigma}. With parameter inherent from PanGu-{\alpha}, we extend the dense Transformer model to sparse one with Random Routed Experts (RRE), and efficiently train the model over 329B tokens by using Expert Computation and Storage Separation(ECSS). This resulted in a 6.3x increase in training throughput through heterogeneous computing. Our experimental findings show that PanGu-{\Sigma} provides state-of-the-art performance in zero-shot learning of various Chinese NLP downstream tasks. Moreover, it demonstrates strong abilities when fine-tuned in application data of open-domain dialogue, question answering, machine translation and code generation.
Forward citations
Cited by 7 Pith papers
-
Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
Constraining transformer projection weights to a shared low-rank subspace reportedly enables near-lossless compression of pipeline-parallel communication, matching centralized convergence at 80Mbps bandwidth.
-
Universal Pansharpening Model
A single pansharpening model works across 4-, 7-, 8-, and 10-band satellite images by projecting arbitrary-band MS data into a fixed latent space and fusing with PAN via a latent diffusion bridge.
-
DEER: Disentangled Mixture of Experts with Instance-Adaptive Routing for Generalizable Machine-Generated Text Detection
DEER, a disentangled mixture-of-experts detector with RL-based instance routing, reports F1 gains of about 1.4 in-domain and 5.3 points out-of-domain over prior MGT detectors.
-
From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI
Autonomous AI becomes dependable when tool use is embedded in persistent workspaces with reusable skills, shifting evaluation from answers to task closure.
-
StackingNet: Collective Inference Across Independent AI Foundation Models
A lightweight weighted-average 'meta-model' over black-box LLM/VLM outputs improves accuracy, reduces bias, and ranks/prunes unreliable models across regression and classification tasks.
-
Containerized In-Storage Processing and Computing-Enabled SSD Disaggregation
The paper presents DockerSSD, a computational SSD that executes Docker containers on firmware via custom NVMe-based Ethernet and virtualized system calls, with simulated and FPGA-prototype evaluations showing speedups...
-
FuxiMT: Sparsifying Large Language Models for Chinese-Centric Multilingual Machine Translation
FuxiMT combines a frozen BLOOMz model with sparse mixture-of-experts layers, Chinese-first pretraining, and curriculum learning to translate into Chinese from 65 languages, with claimed low-resource gains that the pap...
Discussion (0). Continue with ORCID to comment.