REVIEW 10 cited by
Long Range Language Modeling via Gated State Spaces
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
State space models have shown to be effective at modeling long range dependencies, specially on sequence classification tasks. In this work we focus on autoregressive sequence modeling over English books, Github source code and ArXiv mathematics articles. Based on recent developments around the effectiveness of gated activation functions, we propose a new layer named Gated State Space (GSS) and show that it trains significantly faster than the diagonal version of S4 (i.e. DSS) on TPUs, is fairly competitive with several well-tuned Transformer-based baselines and exhibits zero-shot generalization to longer inputs while being straightforward to implement. Finally, we show that leveraging self-attention to model local dependencies improves the performance of GSS even further.
Forward citations
Cited by 10 Pith papers
-
UniSkip-Mamba: A Frequency-Aware State Space Model for Audio-Visual Temporal Forgery Localization
Skip-scanning Mamba with unified audio-visual sequences reaches 63.4% AP@0.95 on LAV-DF and 63.58% mAP on AV-Deepfake1M by regularizing toward low/mid-frequency forgery cues.
-
Partial Ring Scan: Revisiting Scan Order in Vision State Space Models
Ring-based scanning with selective channel routing improves accuracy, speed, and rotation robustness of vision state-space models.
-
MamV2XCalib: V2X-based Target-less Infrastructure Camera Calibration with State Space Model
MamV2XCalib fuses multi-frame vehicle LiDAR projections with roadside camera images, using 4D correlation volumes and Mamba temporal fusion to regress the camera's rotation error.
-
Systolic Array-based Accelerator for Structured State-Space Models
A specialized systolic-array accelerator with a reconfigurable processing element and diagonal dataflow claims 2000x inference speedup over GPUs for S4 and Liquid-S4 state-space models.
-
MambaMap: Online Vectorized HD Map Construction using State Space Model
MambaMap fuses four previous frames of BEV features and instance queries via gated state space layers, beating prior HD map construction methods on nuScenes and Argoverse2.
-
Training-free Token Reduction for Vision Mamba
MTR uses Mamba's timescale parameter Δ as a token importance score to merge unimportant tokens, giving training-free inference speedups with small accuracy loss.
-
Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection
Routing Mamba applies mixture-of-experts to Mamba projection layers with one shared router, reporting perplexity parity with dense Mamba at roughly half the active parameters on 20B-token pretraining.
-
HydraMamba: Multi-Head State Space Model for Global Point Cloud Learning
A state space model based point cloud network with shuffled Hilbert serialization, a convolutional bidirectional S6 branch, and multi-head S6 achieves new top scores on ModelNet40, ShapeNet, S3DIS, and ScanObjectNN.
-
SpectMamba: Integrating Frequency and State Space Models for Enhanced Medical Image Detection
A Mamba-based detector with frequency attention and Hilbert curve scanning edges out several baselines on pneumonia, brain tumor, and fracture detection.
-
ControlMambaIR: Conditional Controls with State-Space Model for Image Restoration
A diffusion image restoration model with a Mamba condition network reports low LPIPS/FID on several benchmarks, but the PSNR losses and internal inconsistencies undermine the stated performance claims.
Discussion (0). Continue with ORCID to comment.