pith. sign in

arxiv: 2606.19946 · v1 · pith:2TJK62UNnew · submitted 2026-06-18 · 💻 cs.CL · cs.LG

GEMS: Geometric Constraints Enable Multi-Semantic Superposition in LLMs

Yu Deng This is my paper

Pith reviewed 2026-06-26 17:33 UTC · model grok-4.3

classification 💻 cs.CL cs.LG
keywords activation steeringmulti-semantic superpositiongeometric constraintsLLMsinference-time interventiondirectional interferencedistributional deviation
0
0 comments X

The pith

Geometric constraints allow LLMs to superpose multiple semantic directions without collapse.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that collapse during multi-semantic superposition in LLMs decomposes into two independent sources: distributional deviation, where additive changes accumulate in norm across layers and push activations outside the training distribution, and directional interference, where non-orthogonal vectors dampen each other. These sources define the constraints any training-free multi-directional method must satisfy. GEMS instantiates the constraints via norm-preserving weighted superposition and targeted attention-pathway injection to counter deviation, plus real-time orthogonalization to counter interference. On GSM8K this preserves 98 percent accuracy when injecting three non-mathematical directions (baseline 92 percent), while unconstrained addition drops to 4 percent; Wikitext-2 sees only a 2.2 percent PPL rise. Layer probes confirm the orthogonalized signals reach the output with retained semantic specificity.

Core claim

The collapse when superposing multiple semantic directions decomposes into distributional deviation and directional interference, which GEMS addresses through norm-preserving weighted superposition, targeted attention-pathway injection, and real-time orthogonalization.

What carries the argument

GEMS, the method that maps the two collapse sources to the three geometric constraints of norm preservation with weighted superposition, attention-pathway targeting, and real-time orthogonalization.

Load-bearing premise

That distributional deviation and directional interference are the primary independently acting sources of collapse and that the listed geometric constraints mitigate them across models and tasks.

What would settle it

A controlled run in which GEMS is applied yet accuracy still collapses on GSM8K or another standard benchmark when three directions are injected.

Figures

Figures reproduced from arXiv: 2606.19946 by Yu Deng.

Figure 1
Figure 1. Figure 1: GEMS architecture. (a) Layer-wise intervention strength is modulated by a Gaussian en￾velope. (b) The GEMS hook selectively intercepts the oproj output in the sequential residual stream, preserving the subsequent MLP factual pathway. (c) Concurrent expert vectors are orthogonalized and fused under a strict norm-preservation constraint. Contributions: 1. Through diagnostic analysis of two failure modes, we … view at source ↗
Figure 2
Figure 2. Figure 2: reveals the internal dynamics: the baseline exhibits smooth norm growth from L0 (2.1) to L31 (84.7), a stable pattern established during training. ActAdd disrupts this pattern within the inter￾vention window, producing a steep norm acceleration that peaks at 3.4× the baseline by L18–L20. This norm surge propagates through subsequent layers, progressively driving activations outside the training distributio… view at source ↗
Figure 4
Figure 4. Figure 4: Directional interference under ActAdd (α = 0.5). Blue: each direction injected individu￾ally. Red: all three simultaneously. The two failure modes, distributional deviation and directional interference, are independent: norm control alone does not prevent mutual dampening, and orthogonalization alone does not pre￾vent norm surge; multi-directional collapse therefore requires constraints at both levels. 3 M… view at source ↗
Figure 5
Figure 5. Figure 5: Probe 2: Context fidelity across L5–L25 (cosine similarity with baseline). B: GEMS Full, [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
read the original abstract

Activation steering controls model behavior by modifying intermediate hidden states at inference time without retraining. Existing methods handle only single-direction injection; when multiple semantic directions are superposed without constraints, the model collapses. We show that this collapse decomposes into two independently acting sources: distributional deviation, where additive perturbations accumulate in norm across layers and drive activations outside the training distribution, and directional interference, where non-orthogonal semantic vectors mutually dampen when superposed. These two sources define the design constraints that any training-free multi-directional intervention must address. As one instantiation of these principles, we propose GEMS, a training-free method that maps each source to a corresponding geometric constraint: norm-preserving weighted superposition and targeted attention-pathway injection for distributional deviation, and real-time orthogonalization for directional interference. On GSM8K, injecting three concurrent non-mathematical directions preserves accuracy at 98% (baseline 92%), while unconstrained addition collapses to 4%; on Wikitext-2, the same injection incurs only 2.2% PPL increase. Component ablation isolates the causal role of each constraint, and layer-level probes confirm that orthogonalized signals survive the FFN pathway and reach the output distribution with semantic specificity. Qualitative steering effects transfer across architectures from 3B to 31B.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims that collapse in multi-directional activation steering decomposes into two independently acting sources—distributional deviation (norm accumulation driving activations out-of-distribution) and directional interference (non-orthogonality causing mutual damping)—and that these define necessary geometric constraints. GEMS instantiates the constraints via norm-preserving weighted superposition plus targeted attention-pathway injection (for deviation) and real-time orthogonalization (for interference). Experiments on GSM8K show three concurrent non-mathematical directions preserve 98% accuracy (vs. 4% unconstrained) and on Wikitext-2 incur only 2.2% PPL rise; component ablations and layer probes are reported to isolate each constraint's role, with qualitative effects transferring across 3B–31B models.

Significance. If the claimed decomposition and independence hold and the constraints generalize, the work would provide a principled, training-free route to multi-semantic steering, addressing a clear limitation of existing single-direction methods. The component ablations and layer-level probes constitute a strength by attempting to isolate causal contributions rather than reporting only end-to-end gains.

major comments (2)
  1. [Abstract, §3, §4] Abstract and §3 (method) / §4 (experiments): the central claim that distributional deviation and directional interference act independently (so that separate geometric fixes compose) is asserted and supported by component ablations, yet no explicit test of non-interaction—additive decomposition, cross-term measurement, or controlled isolation of whether norm deviation modulates the damping from non-orthogonality—is provided. Without such a test the mapping from sources to constraints remains an unverified modeling assumption.
  2. [§4] §4 (GSM8K and Wikitext-2 results): while aggregate numbers are given, the manuscript does not report number of trials, variance, or statistical significance for the 98% vs. 4% and 2.2% PPL figures, making it impossible to assess whether the claimed mitigation is robust or sensitive to prompt sampling.
minor comments (2)
  1. [§3] Notation for the orthogonalization operator and the weighting scheme in the norm-preserving superposition should be defined once in a single equation block rather than re-introduced inline.
  2. [Abstract, §4] The abstract states results transfer across architectures but provides no table or quantitative comparison of steering fidelity or PPL across the 3B–31B range; a compact summary table would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive report. We address each major comment below. The feedback highlights areas where additional evidence and reporting will strengthen the manuscript, and we commit to revisions accordingly.

read point-by-point responses

  1. Referee: [Abstract, §3, §4] Abstract and §3 (method) / §4 (experiments): the central claim that distributional deviation and directional interference act independently (so that separate geometric fixes compose) is asserted and supported by component ablations, yet no explicit test of non-interaction—additive decomposition, cross-term measurement, or controlled isolation of whether norm deviation modulates the damping from non-orthogonality—is provided. Without such a test the mapping from sources to constraints remains an unverified modeling assumption.

    Authors: We acknowledge that the component ablations demonstrate the individual contributions of each constraint but do not include a direct measurement of interaction effects, such as an additive decomposition or cross-term analysis. This leaves the independence assumption as a modeling choice supported indirectly rather than explicitly verified. In the revised manuscript we will add a controlled experiment that compares the joint intervention against the sum of separate interventions and reports any residual interaction term, thereby providing a direct test of non-interaction. revision: yes

  2. Referee: [§4] §4 (GSM8K and Wikitext-2 results): while aggregate numbers are given, the manuscript does not report number of trials, variance, or statistical significance for the 98% vs. 4% and 2.2% PPL figures, making it impossible to assess whether the claimed mitigation is robust or sensitive to prompt sampling.

    Authors: We agree that the absence of trial counts, variance, and significance testing limits assessment of robustness. The reported figures were obtained from 10 independent runs with varied prompt sampling seeds; we will include means, standard deviations, and paired statistical tests in the revised §4 to quantify variability and significance. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper's chain proceeds from an empirical claim (collapse decomposes into distributional deviation and directional interference) to design constraints and the GEMS instantiation, supported by ablations and benchmark results. No equations or steps reduce a reported outcome to a fitted parameter or self-citation by construction; the independence assertion is presented as shown via component isolation rather than presupposed in a definitional loop. The derivation remains self-contained against external benchmarks without load-bearing self-referential reductions.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The abstract does not specify any free parameters, axioms, or invented entities; the geometric constraints are presented as general principles.

pith-pipeline@v0.9.1-grok · 5751 in / 1116 out tokens · 43190 ms · 2026-06-26T17:33:07.763082+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 3 canonical work pages

  1. [1]

    2023 , eprint =

    Steering Language Models With Activation Engineering , author =. 2023 , eprint =

    2023

  2. [2]

    2023 , eprint =

    Steering Llama 2 via Contrastive Activation Addition , author =. 2023 , eprint =

    2023

  3. [3]

    and Wang, Zifan and Mallen, Alex and Basart, Steven and Koyejo, Sanmi and Song, Dawn and Fredrikson, Matt and Kolter, J

    Zou, Andy and Phan, Long and Chen, Sarah and Campbell, James and Guo, Phillip and Ren, Richard and Pan, Alexander and Yin, Xuwang and Mazeika, Mantas and Dombrowski, Ann-Kathrin and Goel, Shashwat and Li, Nathaniel and Byun, Michael J. and Wang, Zifan and Mallen, Alex and Basart, Steven and Koyejo, Sanmi and Song, Dawn and Fredrikson, Matt and Kolter, J. ...

    Pith/arXiv arXiv

  4. [4]

    Thirty-seventh Conference on Neural Information Processing Systems , year =

    Inference-Time Intervention: Eliciting Truthful Answers from a Language Model , author =. Thirty-seventh Conference on Neural Information Processing Systems , year =

  5. [5]

    2025 , eprint =

    Adaptive Multi-Subspace Representation Steering for Attribute Alignment in Large Language Models , author =. 2025 , eprint =

    2025

  6. [6]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , year =

    Multi-Attribute Steering of Language Models via Targeted Intervention , author =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , year =. doi:10.18653/v1/2025.acl-long.1007 , pages =

    work page doi:10.18653/v1/2025.acl-long.1007 2025

  7. [7]

    Findings of the Association for Computational Linguistics: EMNLP 2025 , year =

    Beyond Linear Steering: Unified Multi-Attribute Control for Language Models , author =. Findings of the Association for Computational Linguistics: EMNLP 2025 , year =. doi:10.18653/v1/2025.findings-emnlp.1278 , pages =

    work page doi:10.18653/v1/2025.findings-emnlp.1278 2025

  8. [8]

    Enhancing Instruction Following of

    Kang, Minjae and Kim, Jaehyung , booktitle =. Enhancing Instruction Following of. 2026 , url =

    2026

  9. [9]

    2021 , eprint =

    Training Verifiers to Solve Math Word Problems , author =. 2021 , eprint =

    2021

  10. [10]

    International Conference on Learning Representations , year =

    Pointer Sentinel Mixture Models , author =. International Conference on Learning Representations , year =

  11. [11]

    2022 , eprint =

    Toy Models of Superposition , author =. 2022 , eprint =

    2022

  12. [12]

    Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space

    Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space , author =. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , year =. doi:10.18653/v1/2022.emnlp-main.3 , pages =

    work page doi:10.18653/v1/2022.emnlp-main.3 2022

  13. [13]

    ICML 2025 Workshop on Reliable and Responsible Foundation Models , year =

    The Geometries of Truth Are Orthogonal Across Tasks , author =. ICML 2025 Workshop on Reliable and Responsible Foundation Models , year =

    2025

  14. [14]

    The Fourteenth International Conference on Learning Representations , year =

    From Data Statistics to Feature Geometry: How Correlations Shape Superposition , author =. The Fourteenth International Conference on Learning Representations , year =

  15. [15]

    Advances in Neural Information Processing Systems , year =

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author =. Advances in Neural Information Processing Systems , year =

  16. [16]

    A Geometric Account of Activation Steering through Angle

    Georgii Aparin and Tatiana Gaintseva , year =. A Geometric Account of Activation Steering through Angle. 2606.06735 , archivePrefix =

    Pith/arXiv arXiv