Pith. sign in

REVIEW 10 cited by

The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.13558 v1 pith:5YVXVX5F submitted 2023-12-21 cs.LG cs.AIcs.CLcs.CV

Pratyusha Sharma , Jordan T. Ash , Dipendra Misra This is my paper · ORCID
classification cs.LGcs.AIcs.CLcs.CV
keywords modelslanguagedataincreasinglaserlayer-selectivellmsrank
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Transformer-based Large Language Models (LLMs) have become a fixture in modern machine learning. Correspondingly, significant resources are allocated towards research that aims to further advance this technology, typically resulting in models of increasing size that are trained on increasing amounts of data. This work, however, demonstrates the surprising result that it is often possible to significantly improve the performance of LLMs by selectively removing higher-order components of their weight matrices. This simple intervention, which we call LAyer-SElective Rank reduction (LASER), can be done on a model after training has completed, and requires no additional parameters or data. We show extensive experiments demonstrating the generality of this finding across language models and datasets, and provide in-depth analyses offering insights into both when LASER is effective and the mechanism by which it operates.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SCOPE and SCION: A Benchmark and an Auditable Reference Pipeline for Schema Induction and Fusion from Text

    cs.AI 2026-05 conditional novelty 6.0 of 10

    A 24-dataset benchmark for inducing schema graphs from raw text, plus an auditable LLM-based pipeline that reports the highest scores on the benchmark's four schema-similarity metrics.

  2. ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations

    cs.CL 2025-06 conditional novelty 6.0 of 10

    ProcrustesGPT searches for per-layer orthogonal rotations that make pretrained LLM weights fit Kronecker or GS structured matrices, cutting 14 to 36 percent of parameters without fine-tuning.

  3. Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Dobi-SVD compresses LLMs via differentiable SVD rank selection, IPCA-based weight reconstruction, and quantized storage remapping, reporting competitive perplexity at 40% parameters.

  4. Transformer-Squared: Self-adaptive LLMs

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Transformer-Squared selectively rescales singular values of an LLM's weights with RL-trained expert vectors, then mixes these experts at inference to adapt to unseen tasks.

  5. CURing Large Models: Compression via CUR Decomposition

    cs.LG 2025-01 conditional novelty 6.0 of 10

    CUR decomposition with WANDA-and-DEIM row/column selection compresses LLM weights quickly, and the linking matrix U can be fine-tuned as a PEFT-style healing step.

  6. DeFTX: Denoised Sparse Fine-Tuning for Zero-Shot Cross-Lingual Transfer

    cs.CL 2025-05 conditional novelty 5.0 of 10

    DeFT-X applies SVD denoising to weight updates before magnitude pruning in composable sparse fine-tuning, showing small average gains over LT-SFT on NusaX and AmericasNLI.

  7. Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Field

    cs.DL 2024-12 conditional novelty 5.0 of 10

    Zero-shot LLMs, especially Claude 3 Sonnet and a fine-tuned 7B Mistral variant, classify semantic relations between engineering research topics with high F1 on the new IEEE-Rel-1K benchmark.

  8. KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models

    cs.CL 2024-12 conditional novelty 5.0 of 10

    KaSA improves parameter-efficient fine-tuning by SVD-truncating pretrained weights and learning singular-value parameterized updates, reporting gains over 14 PEFT baselines.

  9. Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models

    cs.LG 2025-01 conditional novelty 4.0 of 10

    A low-rank LLM compression method that stores a row basis of each weight matrix plus coefficients, together with an online error-minimizing reconstruction, reports perplexity near 2:4 semi-structured pruning with bett...

  10. Accelerating Attention with Basis Decomposition

    cs.LG 2025-10 reject novelty 3.0 of 10

    A low-rank factorization of attention projection matrices (basis plus coefficients) gives modest FLOP savings in exact arithmetic, but the claimed losslessness and novelty are not supported.

Pith tools