REVIEW 4 cited by
The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Transformer-based Large Language Models (LLMs) have become a fixture in modern machine learning. Correspondingly, significant resources are allocated towards research that aims to further advance this technology, typically resulting in models of increasing size that are trained on increasing amounts of data. This work, however, demonstrates the surprising result that it is often possible to significantly improve the performance of LLMs by selectively removing higher-order components of their weight matrices. This simple intervention, which we call LAyer-SElective Rank reduction (LASER), can be done on a model after training has completed, and requires no additional parameters or data. We show extensive experiments demonstrating the generality of this finding across language models and datasets, and provide in-depth analyses offering insights into both when LASER is effective and the mechanism by which it operates.
Forward citations
Cited by 4 Pith papers
-
SCOPE and SCION: A Benchmark and an Auditable Reference Pipeline for Schema Induction and Fusion from Text
A 24-dataset benchmark for inducing schema graphs from raw text, plus an auditable LLM-based pipeline that reports the highest scores on the benchmark's four schema-similarity metrics.
-
ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations
ProcrustesGPT searches for per-layer orthogonal rotations that make pretrained LLM weights fit Kronecker or GS structured matrices, cutting 14 to 36 percent of parameters without fine-tuning.
-
DeFTX: Denoised Sparse Fine-Tuning for Zero-Shot Cross-Lingual Transfer
DeFT-X applies SVD denoising to weight updates before magnitude pruning in composable sparse fine-tuning, showing small average gains over LT-SFT on NusaX and AmericasNLI.
-
Accelerating Attention with Basis Decomposition
A low-rank factorization of attention projection matrices (basis plus coefficients) gives modest FLOP savings in exact arithmetic, but the claimed losslessness and novelty are not supported.
Discussion (0). Continue with ORCID to comment.