A noisy top-k gated mixture-of-experts layer between LSTMs scales neural networks to 137B parameters with sub-linear compute, beating SOTA on language modeling and machine translation.
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
4 Pith papers cite this work. Polarity classification is still indexing.
4
Pith papers citing it
abstract
We show that an end-to-end deep learning approach can be used to recognize either English or Mandarin Chinese speech--two vastly different languages. Because it replaces entire pipelines of hand-engineered components with neural networks, end-to-end learning allows us to handle a diverse variety of speech including noisy environments, accents and different languages. Key to our approach is our application of HPC techniques, resulting in a 7x speedup over our previous system. Because of this efficiency, experiments that previously took weeks now run in days. This enables us to iterate more quickly to identify superior architectures and algorithms. As a result, in several cases, our system is competitive with the transcription of human workers when benchmarked on standard datasets. Finally, using a technique called Batch Dispatch with GPUs in the data center, we show that our system can be inexpensively deployed in an online setting, delivering low latency when serving users at scale.
citation-role summary
background 2
citation-polarity summary
roles
background 2polarities
background 2representative citing papers
The paper categorizes five concrete AI safety problems arising from flawed objectives, costly evaluation, and learning dynamics.
EPNAS uses a progressive search policy with REINFORCE performance prediction to search neural architectures in parallel, supporting multiple resource constraints and outperforming ENAS and PNAS on CIFAR-10 and ImageNet in speed and accuracy.
Proposes Tolerance Tiers architecture for MLaaS to let consumers select accuracy-latency trade-offs, shown to outperform single-version deployment on ASR and vision workloads.
citing papers explorer
-
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
A noisy top-k gated mixture-of-experts layer between LSTMs scales neural networks to 137B parameters with sub-linear compute, beating SOTA on language modeling and machine translation.
-
Concrete Problems in AI Safety
The paper categorizes five concrete AI safety problems arising from flawed objectives, costly evaluation, and learning dynamics.
-
EPNAS: Efficient Progressive Neural Architecture Search
EPNAS uses a progressive search policy with REINFORCE performance prediction to search neural architectures in parallel, supporting multiple resource constraints and outperforming ENAS and PNAS on CIFAR-10 and ImageNet in speed and accuracy.
-
One Size Does Not Fit All: Quantifying and Exposing the Accuracy-Latency Trade-off in Machine Learning Cloud Service APIs via Tolerance Tiers
Proposes Tolerance Tiers architecture for MLaaS to let consumers select accuracy-latency trade-offs, shown to outperform single-version deployment on ASR and vision workloads.