REVIEW 6 cited by
Accelerating Transformer Inference for Translation via Parallel Decoding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Autoregressive decoding limits the efficiency of transformers for Machine Translation (MT). The community proposed specific network architectures and learning-based methods to solve this issue, which are expensive and require changes to the MT model, trading inference speed at the cost of the translation quality. In this paper, we propose to address the problem from the point of view of decoding algorithms, as a less explored but rather compelling direction. We propose to reframe the standard greedy autoregressive decoding of MT with a parallel formulation leveraging Jacobi and Gauss-Seidel fixed-point iteration methods for fast inference. This formulation allows to speed up existing models without training or modifications while retaining translation quality. We present three parallel decoding algorithms and test them on different languages and models showing how the parallelization introduces a speedup up to 38% w.r.t. the standard autoregressive decoding and nearly 2x when scaling the method on parallel resources. Finally, we introduce a decoding dependency graph visualizer (DDGviz) that let us see how the model has learned the conditional dependence between tokens and inspect the decoding procedure.
Forward citations
Cited by 6 Pith papers
-
Token Radius Attention for Efficient Video Generation
Video diffusion transformers can run ~1.5-2x faster with competitive quality by converting each query's attention entropy into a spatially decayed retention radius instead of dense attention.
-
Gumbel Distillation for Parallel Text Generation
Conditioning parallel decoders on Gumbel noise sampled from an autoregressive teacher's Gumbel-Max process improves generation quality on LM1B and OpenWebText.
-
StagFormer: Time Staggering Transformer Decoding for RunningLayers In Parallel
StagFormer staggers transformer layers one time step apart with delayed cross-attention, enabling depth-parallel decoding at quality comparable to a deeper baseline.
-
FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharing
A block-pruning and model-extension method for LLMs that replaces pruned blocks with weight-shared blocks plus low-rank adapters, reporting state-of-the-art recovery on several benchmarks.
-
CoT-X: An Adaptive Framework for Cross-Model Chain-of-Thought Transfer and Optimization
An adaptive summarization framework compresses chain-of-thought traces and transfers them across model families, claiming up to 40.5% accuracy gains over truncation on medical QA and 84% fewer configuration evaluation...
-
Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey
A survey that groups VLA efficiency techniques into four categories: model architecture, perception features, action generation, and training/inference strategies.
Discussion (0). Continue with ORCID to comment.