Pith. sign in

REVIEW 5 major objections 5 minor 43 references

High-frequency spectral shift in early SSM kernels explains why code models forget syntax during fine-tuning

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:46 UTC pith:2MUDTDQN

load-bearing objection Solid empirical study of SSM vs Transformer on code, but the spectral-shift explanation is undercut by the paper's own intervention data. the 5 major comments →

arxiv 2602.06774 v2 pith:2MUDTDQN submitted 2026-02-06 cs.AI

Towards Understanding What State Space Models Learn About Code

Jiali Wu , Abhinav Anand , Shweta Verma , Mira Mezini This is my paper
classification cs.AI
keywords state space modelscode understandingkernel analysisspectral shiftinterpretabilityfine-tuningtype inferencehigh-frequency bias
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish why a state space model (SSM) that outperforms a Transformer on code retrieval falls behind on type inference, and to show that the cause is diagnosable and fixable. By analyzing hidden representations and the learned convolution kernels layer by layer, the authors find that during fine-tuning on type inference, the early layers' kernels shift their spectral energy toward high-frequency, short-range dependencies, discarding the long-range context the task needs. They introduce SSM-Interpret, a frequency-domain framework that classifies kernels by spectral centroid and low-to-high frequency energy ratio, and use it to connect this spectral shift to the model's forgetting of syntactic and semantic relations. The claim is actionable: architectural modifications that prevent the shift and add a high-frequency path improve NLCodeSearch MRR from 25.39 to 30.89 and type-inference F1 from 59.70 to 60.98, so the analysis directly informs better model design.

Core claim

The central claim is that CodeSSM, an SSM-based code model, captures syntactic and semantic structures more effectively than a comparable Transformer during pretraining, but during fine-tuning on type inference its early-layer convolution kernels undergo a spectral shift toward high-frequency, short-range frequencies; this shift, combined with only two kernels per layer, makes the model discard long-range token relations and causes it to forget critical code properties. The paper supports this with a kernel analysis framework, SSM-Interpret, that classifies kernels as low-pass (long-range), high-pass (short-range), or band-pass; it shows that complementary forward/backward kernels correlate

What carries the argument

SSM-Interpret, a frequency-domain framework that extracts the forward and backward convolution kernels of each SSM layer, computes their Fourier transforms, and classifies each kernel as low-pass, band-pass, or high-pass using two thresholds: spectral centroid (0.16 and 0.33 of normalized frequency) and low-to-high frequency energy ratio (LHFR, with low band set to bottom 10% and high band to top 40% of the spectrum). The classification reveals complementary versus degenerate kernel pairings, and the authors use this to attribute the type-inference failure to a high-frequency shift in early layers. The architectural fix introduces a parallel high-frequency path using grouped 1D convolutions

Load-bearing premise

The classification of kernels into low-, band-, and high-pass relies on spectral-centroid and LHFR thresholds that the authors tuned after inspecting the very CodeSSM kernels they analyze, so the observed high-frequency shift during fine-tuning could be an artifact of those threshold choices rather than a stable property of the model.

What would settle it

Fine-tune CodeSSM on type inference under several different, reasonable spectral-centroid and LHFR thresholds (e.g., centroid 0.25/0.4, low band 15%, high band 35%) and check whether the early-layer kernels still consistently shift toward high-pass; if the shift disappears or reverses under nearby thresholds, the causal link between spectral shift and forgetting would be unsupported. Alternatively, train CodeSSM-HF with the CNN path removed but the 8-kernel change kept; if that variant shows no improvement over the 1-kernel baseline, the high-frequency inductive bias, not kernel count, is the

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the spectral-shift explanation is correct, then fine-tuning an SSM-based code model on any task that requires both local and global context will tend to degrade its syntactic and semantic representations unless the architecture preserves low-frequency (long-range) sensitivity in early layers.
  • The finding provides a mechanism for why hybrid architectures that combine SSM and Transformer layers outperform pure SSMs: they add a complementary path for high-frequency, local dependencies that a two-kernel SSM cannot supply.
  • The kernel classification framework offers a diagnostic tool for SSM-based models beyond code: practitioners can inspect the spectral centroid and LHFR of learned kernels to anticipate whether a fine-tuning task will induce harmful spectral degeneration.
  • The superior performance of CodeSSM-8kernel over CodeSSM-1024kernel suggests that a moderate number of kernels, each shared across some dimensions, strikes a better balance between capacity and learning coherent, non-redundant frequency responses than either one kernel or one kernel per dimension.
  • If the claims hold, interpretability analysis of convolution kernels can be used as a design constraint rather than a post-hoc explanation, guiding architectural changes that improve downstream accuracy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The spectral-shift mechanism may generalize to other SSM-based models and tasks in NLP and audio, where fine-tuning often changes the effective receptive field; the same spectral centroid and LHFR metrics could predict when a model will fail on tasks requiring long-range reasoning.
  • The threshold choices (spectral centroid 0.16/0.33 and LHFR bands 10%/40%) were selected after inspecting the same kernels that the framework then classifies; a follow-up could compute these thresholds on held-out models or use a fully data-driven clustering approach, and the test would be whether the spectral-shift narrative survives.
  • A testable extension: deliberately regularize or constrain the early-layer kernels to stay low-pass during fine-tuning (e.g., via spectral norm penalties) and measure whether type-inference F1 improves without the parallel CNN path; this would isolate the causal role of the spectral shift from the effect of added parameters.
  • The authors' distinction between low-pass as long-range and high-pass as short-range implies that token distance in code is monotone in frequency, but code structure (AST nesting, variable scopes) may violate this in subtle ways; validating that mapping with synthetic programs of varied nesting and spacing would strengthen the conclusion.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents a comparative analysis of CodeSSM (an S4D-based SSM) and RoCoder (a RoPE Transformer) on code understanding tasks. Using DirectProbe, it reports that pretrained CodeSSM captures AST/DFG relations better than RoCoder, but fine-tuning on type inference causes CodeSSM to forget these relations, especially at short token distances. To explain this, the paper introduces SSM-Interpret, a frequency-domain framework that classifies SSM convolution kernels by spectral centroid and low-to-high frequency energy ratio into low-pass, band-pass, or high-pass filters. It claims that fine-tuning on type inference causes a high-frequency spectral shift in early layers and that this shift is the root cause of CodeSSM's degradation. Based on this diagnosis, it proposes two architectural variants, CodeSSM-HF (with a parallel 1D CNN high-frequency path) and CodeSSM-8kernel, and reports improvements on NLCodeSearch, SQA, and type inference benchmarks.

Significance. If the central diagnostic claim were established, this would be one of the first kernel-level analyses of multi-layer SSMs for code and a valuable example of interpretability-led architecture design. The paper uses classifier-free probing, compares models trained under the same conditions, and the benchmark gains, though modest, are consistent across tasks. However, the significance is currently limited by two load-bearing problems: the spectral classification thresholds are chosen post hoc on the same kernels they are used to explain, and the proposed fix (CodeSSM-HF) does not actually prevent or reverse the high-frequency shift, as the paper itself reports. The contribution would be substantially stronger if the causal claim were reframed around spectral complementarity/diversity rather than high-frequency shift per se, and if the classification thresholds were shown to be robust.

major comments (5)
  1. [§4.2 vs. §5.2] The paper's causal claim is internally inconsistent. §4.2 and the Introduction state that the root cause of CodeSSM's type-inference degradation is a high-frequency spectral shift in early-layer kernels and that the proposed modifications 'prevent this degeneration.' However, §5.2 and Fig. 5d report that CodeSSM-HF has a greater prevalence of high-pass filters in early layers than CodeSSM. The data thus support an alternative explanation: the key is loss of complementary forward/backward spectral coverage or multi-crest spectra, not high-pass behavior per se. Because the fix works while increasing the allegedly harmful quantity, the stated causal mechanism is not supported by the paper's own evidence. Please reframe the central claim or provide a direct test (e.g., a variant that reduces early-layer high-pass prevalence) that separates high-pass prevalence from spectral complementarity.
  2. [§4.1, Eq. (2), Appendix A.2] The kernel classification thresholds are fitted to the same kernels they are used to explain. The spectral centroid thresholds (0.16, 0.33) and LHFR thresholds (10%/40%, ratios >10/<1) are described as 'derived from the observed frequency distribution of CodeSSM kernels' and chosen to 'minimize outliers' across the models under analysis. This makes Fig. 5's categories post hoc and not independent evidence for a spectral shift. Please report a sensitivity analysis over a range of thresholds, or define a priori criteria (e.g., energy majority), and show that the claimed fine-tuning shift is stable. Also provide bootstrap or significance intervals for the shift in classification distributions.
  3. [§5.1, Table 1] The proposed interventions confound the independent variable. CodeSSM-HF adds a 1D CNN path, reduces depth from 12 to 11, changes group size, and changes parameter count; CodeSSM-8kernel changes the number of kernels. These are bundles of architectural changes, so the benchmark gains cannot be attributed to 'preventing the spectral shift.' The subsequent kernel analysis of CodeSSM-HF is performed on the new architecture and thus cannot validate the original causal mechanism. An ablation isolating the spectral property (e.g., a kernel regularizer on high-pass energy, or a CNN path with a low-pass filter) would be needed to make the causal claim.
  4. [Table 1 / Figures 2-4] No uncertainty estimates are reported. The central quantitative claims—the forgetting drop in probing accuracy and the benchmark improvements (e.g., +5.5 MRR, +1.28 F1)—are single point estimates without error bars or significance tests. Given that several differences are small (e.g., type-inference F1 59.70→60.98), reporting multiple seeds and confidence intervals is necessary to establish that the effects are real and not within training noise.
  5. [§4.1, Figure 7] The mapping from high-frequency kernel components to short-range code dependencies is asserted rather than validated. Figure 7 gives a qualitative illustration but no quantitative link. Since this mapping underlies the entire interpretability argument, please validate it (e.g., by correlating kernel classification with distance-binned probing accuracy) or explicitly frame it as a modeling assumption rather than an established fact.
minor comments (5)
  1. [Abstract] Typos: 'forgets certain relations' is ungrammatical; 'upto' should be 'up to'; 'on task' should be 'on a task' or 'on tasks.'
  2. [Eq. (1)] The denominator has a missing closing parenthesis: 'X(f(n)' should be 'X(f(n))'.
  3. [Figure 3 caption] Inconsistent capitalization: 'Rocoder' vs. 'RoCoder.' Also 'Layer 10 and 11 of RoCoder is missing' should be 'are missing.'
  4. [Appendix A.4 / A.7] Grammar issues: 'and and' duplicated in A.4; 'This believe this is the reason' in A.7 should be 'We believe this is the reason.'
  5. [References] Some references have encoding artifacts (e.g., 'L´opez') and there is inconsistent use of accents; please clean up. No URL or artifact link is provided for the SSM-Interpret code; consider releasing it for reproducibility.

Circularity Check

2 steps flagged

Moderate circularity: kernel-classification thresholds are fitted to the same kernels they explain, and the frequency-to-range mapping is a self-cited ansatz; external benchmarks and DirectProbe forgetting remain independent.

specific steps
  1. fitted input called prediction [Eq. 2 and Eq. 4, §4.1, Appendix A.2; applied in §4.2]
    "We established the specific thresholds for both metrics through a rigorous qualitative analysis of the kernel spectral profiles of all the models and our chosen values minimizes the outliers. ... For the CodeSSM model and its variants, the threshold values reported in the paper provided the most consistent and robust classification, minimizing outliers across all experiments."

    The spectral-centroid and LHFR thresholds used to label kernels as low-/band-/high-pass are tuned on the very CodeSSM kernels whose fine-tuning 'spectral shift' they are then used to establish. The shift reported in §4.2 is the output of a classifier whose bins were selected post hoc to make those kernels classify cleanly, so the diagnostic observation is partly manufactured by the measurement instrument rather than an independent finding. Changing the thresholds could alter or remove the claimed shift, meaning the root-cause measurement is not independent of the fitted classification. The DirectProbe forgetting results and benchmark gains are external, but the spectral-shift explanation is partially self-confirming.

  2. self citation load bearing [§4.1, first paragraph; also Appendix A.2]
    "By examining which frequency bands are amplified or attenuated, we classify kernels into low-pass (long-range), high-pass (short-range), or band-pass (mid-range) categories (Ravikumar et al., 2026): Low-pass kernels correspond to long-range dependency capture, whereas high-pass kernels emphasize local dependencies."

    The interpretive step that converts a filter label ('high-pass') into a dependency-range claim ('short-range') is load-bearing: without it, the spectral classifications cannot support the paper's causal story about forgetting short-range relations. This mapping is imported from Ravikumar et al. (2026), a self-citation whose author list overlaps with the present paper (Anand, Verma, Mezini). The paper provides no independent derivation or external validation of this frequency-to-range correspondence for code tokens, so the 'spectral shift toward short-range dependencies' inherits the prior work's labeling rather than being independently established here.

full rationale

The paper's most valuable empirical results are not circular. The DirectProbe measurements in Figs. 2-4 use held-out AST/DFG labels and do not depend on the kernel classifier, and the improvements of CodeSSM-HF and CodeSSM-8kernel are evaluated on external benchmarks (NLCodeSearch, SQA, TypeInference) against CodeSSM and RoCoder. Those parts are self-contained and reproducible in principle. However, the causal explanation has two genuinely circular weak points. First, the SSM-Interpret thresholds (Eqs. 2/4) were explicitly selected by inspecting the same kernels they are used to classify ('our chosen values minimizes the outliers'), so the reported early-layer high-frequency shift is partly a re-description of the classifier's own bins rather than an independent measurement. Second, the mapping from 'high-pass' to 'short-range' is a self-cited assumption from Ravikumar et al. (2026), a same-group paper, and no external validation is supplied. The internal contradiction in §5.2 - CodeSSM-HF shows a greater prevalence of high-pass filters in early layers than CodeSSM yet performs better - further weakens the root-cause attribution, though it is a support problem rather than definitional circularity. Overall, the central claim still has independent content, but the spectral-shift diagnosis is partially constructed from fitted thresholds and self-cited interpretive labels.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The central claims rest on the probing methodology, the frequency-to-dependency-range mapping, the hand-tuned classification thresholds, and the causal interpretation. No fundamentally new physical or mathematical entities are introduced; the main invented component is an architectural module with direct benchmark evidence.

free parameters (4)
  • Spectral centroid low/high thresholds = SC < 0.16 low-pass; SC > 0.33 high-pass
    Eq. (2), Section 4.1: 'thresholds, derived from the observed frequency distribution of CodeSSM kernels'; chosen to minimize outliers (A.2).
  • LHFR low/high frequency bands and thresholds = low = bottom 10%, high = top 40%; low-pass if LHFR ≫ 10, high-pass if LHFR ≪ 1
    Eq. (3)-(4) and Appendix A.2: thresholds obtained by qualitative inspection 'yielding no outliers across all CodeSSM model kernels'.
  • CodeSSM-HF CNN kernel size and group size = kernel_size=3, group_size=8, depth reduced to 11 layers
    Section 5.1 and Table 2: group 8 selected over 4; kernel size 3 chosen for local 3-token context.
  • Number of SSM kernels per layer in CodeSSM-8kernel = 8 kernels per layer, shared across 128 dimensions
    Section 5.1: chosen from pretraining sweep over 1,4,8,...,512,1024 kernels on a small dataset; 8 gave best pretraining performance.
axioms (4)
  • domain assumption The Fourier transform of learned SSM convolution kernels yields a meaningful spectral decomposition of the token-dependency range the model captures.
    Section 4.1 and Figure 7 map low/high frequency to long/short-range code relationships; this mapping is asserted, not derived.
  • domain assumption DirectProbe cluster accuracy measures how well the hidden representation encodes a code property.
    Section 3.2; inherited from Anand et al. (2024) and Zhou & Srikumar (2021), treated as a valid probe of learned structure.
  • ad hoc to paper The high-frequency shift observed in early layers during fine-tuning on type inference is causal, not merely correlational, in producing the forgetting.
    Section 4.2 and the contributions list call it the 'root cause'; no intervention isolating the shift from other fine-tuning changes is performed.
  • ad hoc to paper Multiplying forward and backward SSM outputs in the time domain, when both kernels are high-pass, amplifies the high-frequency response.
    Section 4.2 invokes time-domain multiplication ⇔ frequency-domain convolution; the specific amplification claim for arbitrary high-pass kernels is not proven and is not a standard theorem.
invented entities (1)
  • CodeSSM-HF high-frequency parallel path (1D CNN, kernel size 3, grouped convolution) independent evidence
    purpose: Add explicit local/short-range inductive bias parallel to the SSM block to counteract the fine-tuning spectral shift.
    The component is tested on NLCodeSearch, SQA, and type-inference (Table 1); however it is not derived from the analysis and is one of several ad hoc fixes (HF, 1024kernel, 8kernel) with mixed results.

pith-pipeline@v1.3.0-alltime-deepseek · 20004 in / 14161 out tokens · 121641 ms · 2026-08-03T03:46:48.474428+00:00 · methodology

0 comments
read the original abstract

State Space Models (SSMs) have emerged as an efficient alternative to the Transformer architecture. Prior work shows that, when trained under comparable conditions, SSMs can match or surpass Transformers on code understanding tasks. However, their internal mechanisms remain a black box. We present the first systematic analysis of what SSM-based code models learn along with the direct comparison between SSM and Transformer models in this domain. Our analysis shows that SSMs capture syntactic and semantic structure more effectively than Transformers during pretraining but forgets certain relations during fine-tuning on some tasks. To investigate this behavior, we introduce SSM-Interpret, a frequency-domain framework that exposes a spectral shift toward short-range dependencies during fine-tuning. Guided by these findings, we propose architectural modifications that significantly improve the performance of SSM-based code model by upto +6 MRR on NLCodeSearch. This demonstrates that our analysis not only explains model behavior but also leads directly to better designs.

Figures

Figures reproduced from arXiv: 2602.06774 by Abhinav Anand, Jiali Wu, Mira Mezini, Shweta Verma.

Figure 1
Figure 1. Figure 1: The CodeSSM layer architecture (left) showing the original routing mechanism (center) and the proposed routing (right). 3 4 6 8 10 11 layer 0.52 0.54 0.56 0.58 0.60 0.62 mean accuracy across labels 0.590 0.603 0.621 0.623 0.632 0.626 0.559 0.574 0.595 0.581 0.515 codeSSM rocoder 3 4 6 8 10 11 layer 0.700 0.725 0.750 0.775 0.800 0.825 0.850 mean accuracy across labels 0.817 0.819 0.837 0.854 0.847 0.831 0.7… view at source ↗
Figure 2
Figure 2. Figure 2 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Result of hidden representation analysis for CodeSSM and RoCoder along with their fine-tuned versions in terms of mean accuracy across task labels. The left row shows the performance on distance prediction task, center shows performance on sibling prediction task and right shows it on edge prediction task. Layer 10 and 11 of RoCoder is missing because the clustering algorithm does not converge. 2.0 3.0 4.0… view at source ↗
Figure 4
Figure 4. Figure 4: Accuracy of CodeSSM, CodeSSM-typeinf, Rocoder and Rocoder-typeinf on distance prediction tasks for layers 6 and 10. This task measures the model’s grasp of local syntactic relationships. AST Distance Prediction: Estimates the shortest path length between two tokens in the AST. This serves as a proxy for short and long-range syntactic understanding and the model’s understanding of program flow (Anand et al.… view at source ↗
Figure 5
Figure 5. Figure 5: Layer wise filter classification of forward (left) and backward kernel (right) of CodeSSM and its variants. frequencies, thus such regularization is unnecessary. 5. Analysis-Driven Improvements for CodeSSM Our analysis of the hidden representations and SSM ker￾nels has uncovered distinct strengths and critical limitations within the CodeSSM architecture. Leveraging these insights, we propose targeted archi… view at source ↗
Figure 6
Figure 6. Figure 6: SSM Kernel of layer 2 forward (left) and layer 3 back￾ward (right). The dominant frequency is represented with a circle (red) and the spectral centroid is represented by a triangle (green) Second, the final two layers of CodeSSM-HF shift pre￾dominantly toward high-pass behavior. This contrasts with CodeSSM, where later layers show low- or band-pass char￾acteristics. This shift toward high-pass behavior in … view at source ↗
Figure 7
Figure 7. Figure 7: Relationship between frequency ranges and code patterns quency. While LHFR can depend on the choice of cutoff or median frequency, we found that using the median often introduced numerous outliers. To address this, we qualita￾tively analyzed the kernels and experimented with multiple threshold values. The most stable results—yielding no out￾liers across all CodeSSM model kernels—were achieved by defining l… view at source ↗
Figure 8
Figure 8. Figure 8: Filter classification of each layer of the CodeSSM on the basis of spectral centroid and LHFR. The training on git issues and code follows Verma et al. (2025) but the training on Wikipedia data is significantly less than that of BiGS Wang et al. (2023). This results in a slightly lower performance of CodeSSM trained by us compared to Verma et al. (2025). Similar to Verma et al. (2025), we used the CodeT5pl… view at source ↗
Figure 9
Figure 9. Figure 9: Visual representation of DirectProbe. (for example, 1024), there is a higher chance of out of phase kernels in forward and backward paths which can cancel out the encoded token relations when the hidden representation of forward and backward paths are multiplied (equivalent to convolution in frequency domain). This believe this is the reason for lower performance of CodeSSM-1024kernel. 15 [PITH_FULL_IMAGE… view at source ↗
Figure 10
Figure 10. Figure 10: Comparative analysis of hidden representation of CodeSSM and RoCoder (left), after finetuning on SQA (center) and after finetuning on type inference (right) for some layers. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Additional kernel visualization showing kernels of layer 1 (left), layer 5 (center) and the last layer (right). In each sub-figure top row shows forward kernels and bottom row shows backward kernels. The visualization shows the richer kernels learnt by CodeSSM-HF. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: The forward and backward kernels of layer 1 of CodeSSM-8kernel model. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 6 canonical work pages · 1 internal anchor

  1. [1]

    A., Hirata, R., and Wang, Z

    Abello, A. A., Hirata, R., and Wang, Z. Dissecting the high-frequency bias in convolutional neural networks. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp.\ 863--871, 2021. doi:10.1109/CVPRW53098.2021.00096

    arXiv 2021

  2. [2]

    Towards Understanding What Code Language Models Learned

    Ahmed, T., Yu, D., Huang, C., Wang, C., Devanbu, P., and Sagae, K. Towards understanding what code language models learned. CoRR, abs/2306.11943, 2023. doi:10.48550/arXiv.2306.11943. URL https://doi.org/10.48550/arXiv.2306.11943

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2306.11943 2023

  3. [3]

    A., Zimerman, I., and Wolf, L

    Ali, A. A., Zimerman, I., and Wolf, L. The hidden attention of mamba models. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T. (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1516--1534, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89...

    doi:10.18653/v1/2025.acl-long.76 2025

  4. [4]

    A critical study of what code- LLM s (do not) learn

    Anand, A., Verma, S., Narasimhan, K., and Mezini, M. A critical study of what code- LLM s (do not) learn. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp.\ 15869--15889, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi:10.18653/v1/2024.findings-acl.93...

    doi:10.18653/v1/2024.findings-acl.939 2024

  5. [5]

    Zoology: Measuring and improving recall in efficient language models, 2023

    Arora, S., Eyuboglu, S., Timalsina, A., Johnson, I., Poli, M., Zou, J., Rudra, A., and Ré, C. Zoology: Measuring and improving recall in efficient language models, 2023. URL https://arxiv.org/abs/2312.04927

    Pith/arXiv arXiv 2023

  6. [6]

    Probing classifiers: Promises, shortcomings, and advances

    Belinkov, Y. Probing classifiers: Promises, shortcomings, and advances. Comput. Linguistics, 48 0 (1): 0 207--219, 2022. doi:10.1162/coli\_a\_00422. URL https://doi.org/10.1162/coli\_a\_00422

    doi:10.1162/coli 2022

  7. [7]

    Blackledge, J. M. Digital Signal Processing: Mathematical And Computational Methods, Software Development And Applications (Second Edition). Horwood Publishing Limited, 2006. ISBN 1904275265

    2006

  8. [8]

    Chen, T., Lin, P., Wang, Z., and Xu, Z.-Q. J. Achilles' heel of mamba: Essential difficulties of the mamba architecture demonstrated by synthetic data. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=4p28lkk44b

    2025

  9. [9]

    and Brad, R

    Constantinescu, C. and Brad, R. An overview on sound features in time and frequency domain. International Journal of Advanced Statistics and IT&C for Economics and Life Sciences, 13: 0 45--58, 12 2023. doi:10.2478/ijasitels-2023-0006

    work page doi:10.2478/ijasitels-2023-0006 2023

  10. [10]

    and Gu, A

    Dao, T. and Gu, A. Transformers are ssms: generalized models and efficient algorithms through structured state space duality. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    2024

  11. [11]

    Combining recurrent, convolutional, and continuous-time models with linear state space layers

    Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., and R\' e , C. Combining recurrent, convolutional, and continuous-time models with linear state space layers. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 572--585, 2021. URL https://proceed...

    2021

  12. [12]

    On the parameterization and initialization of diagonal state space models

    Gu, A., Goel, K., Gupta, A., and R\' e , C. On the parameterization and initialization of diagonal state space models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 35971--35983. Curran Associates, Inc., 2022 a . URL https://openreview.net/forum?id=yJE7iQSAep

    2022

  13. [13]

    Efficiently modeling long sequences with structured state spaces

    Gu, A., Goel, K., and Re, C. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022 b . URL https://openreview.net/forum?id=uYLFoz1vlAC

    2022

  14. [14]

    and Liang, P

    Hewitt, J. and Liang, P. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019 , pp.\ 2733--2743. Association for Computational Linguist...

    doi:10.18653/v1/d19-1275 2019

  15. [15]

    Fourier position embedding: Enhancing attention s periodic extension for length generalization

    Hua, E., Jiang, C., Lv, X., Zhang, K., Sun, Y., Fan, Y., Zhu, X., Qi, B., Ding, N., and Zhou, B. Fourier position embedding: Enhancing attention s periodic extension for length generalization. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=ZfDNDkg7Dh

    2025

  16. [16]

    R., Montavon, G., Muller, K

    Jafari, F. R., Montavon, G., Muller, K. R., and Eberle, O. Mamba LRP : Explaining selective state space sequence models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=2n1Ysn1EDl

    2024

  17. [17]

    M., and Malach, E

    Jelassi, S., Brandfonbrener, D., Kakade, S. M., and Malach, E. Repeat after me: Transformers are better than state space models at copying, 2024. URL https://arxiv.org/abs/2402.01032

    Pith/arXiv arXiv 2024

  18. [18]

    and Robbes, R

    Karmakar, A. and Robbes, R. What do pre-trained code models know about code? In 36th IEEE/ACM International Conference on Automated Software Engineering, ASE 2021, Melbourne, Australia, November 15-19, 2021 , pp.\ 1332--1336. IEEE , 2021. doi:10.1109/ASE51524.2021.9678927. URL https://doi.org/10.1109/ASE51524.2021.9678927

    arXiv 2021

  19. [19]

    B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., LI, J., Chim, J., Liu, Q., Zheltonozhskii, E., Zhuo, T

    Li, R., allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., LI, J., Chim, J., Liu, Q., Zheltonozhskii, E., Zhuo, T. Y., Wang, T., Dehaene, O., Lamy-Poirier, J., Monteiro, J., Gontier, N., Yee, M.-H., Umapathi, L. K., Zhu, J., Lipkin, B., Oblokulov, M., Wang, Z., Murthy, R., Stillerman, J. T., Patel, S. S., Abulkhanov, D., ...

    2023

  20. [20]

    L \' o pez, J. A. H., Weyssow, M., Cuadrado, J. S., and Sahraoui, H. A. Ast-probe: Recovering abstract syntax trees from hidden representations of pre-trained language models. In 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, Rochester, MI, USA, October 10-14, 2022 , pp.\ 11:1--11:11. ACM , 2022. doi:10.1145/3551349.35...

    arXiv 2022

  21. [21]

    Frequency regularization: Unveiling the spectral inductive bias of deep neural networks, 2025

    Lu, J. Frequency regularization: Unveiling the spectral inductive bias of deep neural networks, 2025. URL https://arxiv.org/abs/2512.22192

    arXiv 2025

  22. [22]

    K., Fu, S., and LIU, S

    Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., Li, G., Zhou, L., Shou, L., Zhou, L., Tufano, M., GONG, M., Zhou, M., Duan, N., Sundaresan, N., Deng, S. K., Fu, S., and LIU, S. Code XGLUE : A machine learning benchmark dataset for code understanding and generation. In Thirty-fifth Conference ...

    2021

  23. [23]

    L., Fickus, M., Bryan, E., Petkie, D

    Massar, M. L., Fickus, M., Bryan, E., Petkie, D. T., and Terzuoli, A. J. Fast computation of spectral centroids. Adv. Comput. Math., 35 0 (1): 0 83–97, July 2011. ISSN 1019-7168. doi:10.1007/s10444-010-9167-y. URL https://doi.org/10.1007/s10444-010-9167-y

    work page doi:10.1007/s10444-010-9167-y 2011

  24. [24]

    H., Valvoda, J., Pimentel, T., Williams, A., and Cotterell, R

    Maudslay, R. H., Valvoda, J., Pimentel, T., Williams, A., and Cotterell, R. A tale of a probe and a parser. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020 , pp.\ 7389--7395. Association for Computational Linguistics, 2020. doi:10.18653/v1/2020.acl-main.659. URL https://doi.org/...

    doi:10.18653/v1/2020.acl-main.659 2020

  25. [25]

    The illusion of state in state-space models

    Merrill, W., Petty, J., and Sabharwal, A. The illusion of state in state-space models. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    2024

  26. [26]

    W., Shah, P., Dao, T., Baccus, S

    Nguyen, E., Goel, K., Gu, A., Downs, G. W., Shah, P., Dao, T., Baccus, S. A., and R\' e , C. S4nd: modeling images and videos as multidimensional signals using state spaces. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088

    2022

  27. [27]

    and Suzuki, T

    Nishikawa, N. and Suzuki, T. State space models are provably comparable to transformers in dynamic token selection. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=QFgbJOYJSE

    2025

  28. [28]

    Does transformer interpretability transfer to rnns?, 2024

    Paulo, G., Marshall, T., and Belrose, N. Does transformer interpretability transfer to rnns?, 2024. URL https://arxiv.org/abs/2404.05971

    Pith/arXiv arXiv 2024

  29. [29]

    SMR : State memory replay for long sequence modeling

    Qi, B., Gao, J., Zhang, K., Li, D., Liu, J., Wu, L., and Zhou, B. SMR : State memory replay for long sequence modeling. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp.\ 8102--8116, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi:10.18653/v1/2024.fin...

    work page doi:10.18653/v1/2024.findings-acl.483 2024

  30. [30]

    Analysis of long range dependency understanding in state space models, 2026

    Ravikumar, S., Anand, A., Verma, S., and Mezini, M. Analysis of long range dependency understanding in state space models, 2026. URL https://arxiv.org/abs/2601.13048

    arXiv 2026

  31. [31]

    Samba: Simple hybrid state space models for efficient unlimited context language modeling

    Ren, L., Liu, Y., Lu, Y., yelong shen, Liang, C., and Chen, W. Samba: Simple hybrid state space models for efficient unlimited context language modeling. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=bIlnpVM4bc

    2025

  32. [32]

    Roformer: Enhanced transformer with rotary position embedding

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomput., 568 0 (C), March 2024. ISSN 0925-2312. doi:10.1016/j.neucom.2023.127063. URL https://doi.org/10.1016/j.neucom.2023.127063

    arXiv 2024

  33. [33]

    and Jin'no, K

    Togawa, S. and Jin'no, K. Understanding convolutional neural networks through z-transform: Frequency domain analysis of kernel feature extraction. Nonlinear Theory and Its Applications, IEICE, 16 0 (4): 0 878--895, 2025. doi:10.1587/nolta.16.878

    work page doi:10.1587/nolta.16.878 2025

  34. [34]

    C ode SSM : Towards state space models for code understanding

    Verma, S., Anand, A., and Mezini, M. C ode SSM : Towards state space models for code understanding. In Christodoulopoulos, C., Chakraborty, T., Rose, C., and Peng, V. (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp.\ 34219--34235, Suzhou, China, November 2025. Association for Computational Linguistics. IS...

    work page doi:10.18653/v1/2025.emnlp-main.1735 2025

  35. [35]

    What do they capture? a structural analysis of pre-trained language models for source code

    Wan, Y., Zhao, W., Zhang, H., Sui, Y., Xu, G., and Jin, H. What do they capture? a structural analysis of pre-trained language models for source code. In Proceedings of the 44th International Conference on Software Engineering, ICSE '22, pp.\ 2377–2388, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392211. doi:10.1145/3510003.3...

    arXiv 2022

  36. [36]

    N., Gu, A., and Rush, A

    Wang, J., Yan, J. N., Gu, A., and Rush, A. Pretraining without attention. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 58--69, Singapore, December 2023. Association for Computational Linguistics. doi:10.18653/v1/2023.findings-emnlp.5. URL https://aclanthology.org/2023.findings-emnlp.5/

    doi:10.18653/v1/2023.findings-emnlp.5 2023

  37. [37]

    and Li, Q

    Wang, S. and Li, Q. Stablessm: alleviating the curse of memory in state-space models through stable reparameterization. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    2024

  38. [38]

    State space models are strong text rerankers, 2025

    Xu, Z., Yan, J., Gupta, A., and Srikumar, V. State space models are strong text rerankers, 2025. URL https://arxiv.org/abs/2412.14354

    Pith/arXiv arXiv 2025

  39. [39]

    An extensive study of the structure features in transformer-based code semantic summarization

    Yang, K., Mao, X., Wang, S., Qin, Y., Zhang, T., Lu, Y., and Al - Sabahi, K. An extensive study of the structure features in transformer-based code semantic summarization. In 31st IEEE/ACM International Conference on Program Comprehension, ICPC 2023, Melbourne, Australia, May 15-16, 2023 , pp.\ 89--100. IEEE , 2023. doi:10.1109/ICPC58990.2023.00024. URL h...

    arXiv 2023

  40. [40]

    and Srikumar, V

    Zhou, Y. and Srikumar, V. Directprobe: Studying representations without classifiers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021 , pp.\ 5070--5083. Association for Computational Linguistics, 2021. doi:10.18653/v1/2...

    doi:10.18653/v1/2021.naacl-main.401 2021

  41. [41]

    Explaining modern gated-linear rnns via a unified implicit attention formulation, 2024

    Zimerman, I., Ali, A., and Wolf, L. Explaining modern gated-linear rnns via a unified implicit attention formulation, 2024. URL https://arxiv.org/abs/2405.16504

    Pith/arXiv arXiv 2024

  42. [42]

    X., Manavoglu, E., Zhao, T., and Gao, J

    Zuo, S., Liu, X., Jiao, J., Charles, D. X., Manavoglu, E., Zhao, T., and Gao, J. Efficient hybrid long sequence modeling with state space augmented transformers. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=uUIFTjBREk

    2024

  43. [43]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...