Pith. sign in

REVIEW 4 major objections 5 minor 70 references

Riemann GeoResolver: A Non-Euclidean Attention Framework from Euclidean Resolver to Hyperbolic-Spherical Geometry

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that inverse-distance attention beats softmax in expressiveness, optimization, and generalization, and extends the claim to hyperbolic and spherical geometries.

desk verdict The paper's central PL theorem is false due to a bad derivative computation, so the exponential optimization advantage and the non-Euclidean extensions built on it are unsupported; desk reject unless substantially revised. read the letter →

arxiv 2608.10416 v1 pith:IRAAAYII submitted 2026-08-11 cs.DS cs.AIcs.CLcs.LG

Liangchen Ge This is my paper · ORCID
classification cs.DScs.AIcs.CLcs.LG
keywords inverse-distanceattentionPolyak-Lojasiewiczinequalitysoftmaxeffectiverankhyperbolicgeometrysphericalroutingcircuitcomplexityoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to prove that inverse-distance attention—which weights tokens by the inverse of squared distance rather than by softmax over inner products—is a fundamentally stronger attention kernel than softmax. In Euclidean space, it claims three advantages: exact retrieval with only $O(1)$ width where softmax needs $\Omega((\log n)^2)$; a Polyak–Lojasiewicz constant exponentially larger than softmax's for separated keys; and a width-independent effective rank that prevents memorizing arbitrary noisy labels. The second half extends the same inverse-distance principle to hyperbolic geodesic distance for storage and spherical geodesic distance for routing, yielding a ten-module framework with complexity, regret, and routing-quality bounds. If the Euclidean theorems hold, the choice of kernel—not the surrounding architecture—is the source of these guarantees.

What carries the argument

The object carrying the argument is the inverse-distance attention kernel $W_{ij} = (d(q_i,k_j)^2 + \varepsilon)^{-1} / \sum_m (d(q_i,k_m)^2 + \varepsilon)^{-1}$, where $d$ is Euclidean, hyperbolic geodesic, or spherical geodesic distance. In the one-dimensional two-key loss, the proof expands the target weight $W_1(q)$ around $q=0$ at the exact-match key; the assumption $W'_1(0)=0$ produces the quadratic leading term from which the PL constant $\Theta(\varepsilon^2/\Delta^4)$ is extracted. The same expansion, with $\Delta$ replaced by a hyperbolic distance $\Delta_H$ or a spherical angle $\theta$, is reused for the non-Euclidean PL theorems.

What would settle it

For two keys at $0$ and $\Delta$, differentiate $W_1(q) = \frac{(q^2+\varepsilon)^{-1}}{(q^2+\varepsilon)^{-1} + ((q-\Delta)^2+\varepsilon)^{-1}}$ at $q=0$. Direct computation gives $W'_1(0) = -\frac{2\Delta}{\varepsilon(\Delta^2+\varepsilon)^2(\varepsilon^{-1}+(\Delta^2+\varepsilon)^{-1})^2}$, which is nonzero for $\Delta>0$. Finding this nonzero value refutes the expansion after the definitions of $A(q)$ and $B(q)$ in Section 2.6 and, with it, the stated PL constant and the no-spurious-minima corollary.

Watch

Extended reading notes

Core claim

The paper's central claim is that replacing the softmax kernel with an inverse-distance kernel turns attention into an exact retrieval mechanism with favorable optimization geometry. In the Euclidean prototype, Resolver, the three core theorems assert: (1) circuit separation—IDA retrieves an exact match with $O(1)$ resources while any softmax architecture needs $\Omega((\log n)^2)$ width; (2) a Polyak–Lojasiewicz inequality for the one-dimensional two-key loss with $\mu_{\mathrm{IDA}} = \Theta(\varepsilon^2/\Delta^4)$ versus $\mu_{\mathrm{soft}} = \Theta(e^{-\Delta^2/\sqrt{d}}\varepsilon^2/\Delta^2)$, implying linear convergence, $\Theta(1)$ Hessian spread, and no spurious local minima; and (3) a width-independent effective-rank bound $\mathrm{eff\text{-}rank}(K) \le 1 + n\varepsilon^2/d_{\min}^4$, so IDA limits test error to $O(\eta^2)$ under symmetric label noise while softmax memorizes arbitrary labels once the hidden dimension reaches $n$. The non-Euclidean part transfers these results to hyperbolic and spherical geodesic distances using the same inverse-distance kernel and analogous expansions, and adds modules for compression, gating, dynamic memory, and sparse routing.

Load-bearing premise

Theorem 2.6 rests on the claim that the inverse-distance weight $W_1(q)$ has zero first derivative at the exact-match query $q=0$; direct differentiation of the two-key weight gives a nonzero value for separated keys, and without $W'_1(0)=0$ the quadratic expansion and the $\Theta(\varepsilon^2/\Delta^4)$ PL constant do not follow.

Editorial extensions

If this is right

  • Exact-match retrieval would require only $O(1)$ width for IDA, whereas softmax needs $\Omega((\log n)^2)$ width to approximate the same behavior.
  • Under the two-key loss, gradient descent on IDA would converge linearly with rate constant $\Theta(\varepsilon^2/\Delta^4)$, exponentially larger than softmax's constant for well-separated keys, and every stationary point would be either a global minimum or a strict saddle.
  • IDA's effective rank would stay bounded independent of hidden width, so under symmetric label noise the test error would be $O(\eta^2)$ instead of softmax's memorization catastrophe at $d_h \ge n$.
  • Replacing Euclidean distance with hyperbolic distance for storage and spherical distance for routing would preserve the exact-retrieval and PL-style guarantees, with per-token complexity ranging from $\Theta(n^2)$ to $\Theta(1)$ depending on the HIDA variant.
  • Dynamic prototype allocation would achieve $O(\log T)$ regret and sparse spherical routing would have communication cost independent of batch size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: The expressiveness and effective-rank results in Theorems 1 and 3 do not rely on the $W'_1(0)=0$ expansion; if the PL claim fails, those two advantages may still stand on their own.
  • Because the hyperbolic and spherical PL theorems are proved by analogous two-key expansions, the correctness of the non-Euclidean optimization claims is coupled to the Euclidean expansion; a fix or refutation of the Euclidean proof automatically carries over to the non-Euclidean settings.
  • A testable extension is a synthetic two-key optimization with small $\varepsilon$ and separated keys: the paper predicts linear convergence with constant $\Theta(\varepsilon^2/\Delta^4)$, which is directly measurable without building a full attention architecture.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes inverse-distance attention (IDA) as an alternative to softmax attention and claims three Euclidean theorems: circuit separation with O(1) versus Ω((log n)^2) resources, a Polyak–Łojasiewicz inequality with an exponentially larger constant than softmax, and a width-independent effective rank bound preventing noise memorization. It then extends the framework to hyperbolic and spherical geometries, packaging the results into a ten-module architecture (HIDA, HCC, HyperGate, SIDA, DMG, GSR). The Euclidean part is claimed to be proved in full; the non-Euclidean results are stated as following by analogous arguments.

Significance. If correct, the claimed exponential gap in PL constants and the width-independent capacity bounds would be significant for attention optimization and generalization. The manuscript contains some sound elementary observations: the exact-retrieval limit in Lemma 2.1 is straightforward, the softmax lower bound in Theorem 2.2 is a valid and simple calculation for the constructed orthonormal instance, and the complexity counts for FP-HIDA and L-HIDA are checkable. However, the central optimization theorem is false as stated, and the effective-rank claim is internally inconsistent; these are load-bearing results on which the abstract, the summary table, and the non-Euclidean extensions all rest. The paper therefore does not currently establish its main claims.

major comments (4)
  1. [Section 2.6, Theorem 2.6] The proof of the IDA PL inequality is invalid. The expansion asserts W'_1(0)=0, but direct differentiation of W_1(q)=A(q)/(A(q)+B(q)) with A(q)=(q^2+ε)^{-1} and B(q)=((q-Δ)^2+ε)^{-1} gives W'_1(0)=-A(0)B'(0)/(A(0)+B(0))^2 = -2Δ/(ε(Δ^2+ε)^2(1/ε+1/(Δ^2+ε))^2), which is nonzero for Δ>0. The local quadratic behavior L(q)-L(0)≍q^2 therefore does not hold; the leading term is linear. More decisively, W'_1(q)=0 at the two roots of q^2-Δq-ε=0, and for the generic instance v1=1, v2=0, y=1/2 the loss L(q)=(W_1(q)-1/2)^2 is positive at both stationary points while inf_q L(q)=0. Hence the global PL inequality 2μ(L-L*)≤|L'|^2 fails at those stationary points for every μ>0. This invalidates the stated constant μ_IDA=Θ(ε^2/Δ^4), the ratio μ_IDA/μ_soft, the linear-convergence claim, and Corollary 2.7's absence-of-spurious-minima statement. The softmax PL constant is also not a global PL constant, since for the same 1D two-key loss the gradient tends to zero as q→±∞ while L-L* tends to 1/4.
  2. [Section 2.5, Lemma 2.5] The lemma is internally contradictory. The statement claims ∂²L/∂δ²|₀=Θ(1) for IDA, independent of n, but the proof computes W''₁₁(0)≈-2(n-1)/ρ² and then concludes L''(0)=Θ(W''₁₁(0)), which is Θ(n) in the stated regime. The subsequent sentence asserts Θ(1) Hessian spread, but that does not repair the contradiction in the second-derivative magnitude. Since Corollary 2.7 explicitly invokes Lemma 2 for the Θ(1) Hessian spread and saddle-escape claim, this part of the optimization story is unsupported.
  3. [Section 2.7, Theorem 2.8] The refined effective-rank bound is arithmetically inconsistent with the proof's own computation. The proof derives eff-rank(K)≤n/(1+ε²(n-1)/d_min⁴), which for small ε is close to n, while the theorem claims eff-rank(K)≤1+nε²/d_min⁴. For a concrete example with n=100, ε=0.01, d_min=1, the stated bound is roughly 1.01, whereas the displayed formula gives roughly 99. The refined bound cannot follow from the same effective-rank expression, so the width-independent capacity claim, and with it the noise-memorization contrast in Theorem 2.10, lacks a valid proof.
  4. [Theorems 5.3 and 8.1] The non-Euclidean PL claims are not independent results: both proofs reuse the same expansion W'_1(0)=0 from Theorem 2.6. In the spherical case the weight W_1(t) has exactly the same functional form with Δ=θ, so the nonzero derivative at t=0 applies verbatim and the claimed PL constant Θ(ε²/θ⁴) is unsupported. In the hyperbolic case the same flawed step is imported through the 'same expansion as Theorem 2' argument. The paper's stated limitation that non-Euclidean theorems are proved by analogous arguments is therefore not a harmless presentation choice; it propagates the central error to the entire Part II.
minor comments (5)
  1. [References] Reference [13] (Ge, 2026, 'Under review') is listed but never cited in the text. Since it appears to be the author's own companion manuscript, it should either be cited where the Euclidean results are used or removed.
  2. [Section 2.7 and throughout] The notation for the smoothing constant is inconsistent: the text uses both ε and ϵ, sometimes within the same theorem statement, as in Theorem 2.8. Please standardize.
  3. [Section 2.4, Lemma 2.4] Step 4 of the proof states that 'the number of non-negligible distance scales is O(log n)' without a precise definition or proof; either supply the missing argument or state the clustering assumption in a way that makes this step checkable.
  4. [Sections 5 and 11.4] Several cross-references are wrong: Section 5 refers to 'Lemma 0' and Section 11.4 refers to 'Proposition 2', but no Lemma 0 or Proposition 2 exists in the paper. These should be corrected to the actual lemma and proposition numbers.
  5. [Section 9, Lemma 9.1] The supermartingale argument for the adaptive threshold regret bound is only sketched; the claim that the trigger condition creates negative drift and yields E[S_T]=O(log T) needs a formal proof, especially regarding the dependence of σ_t on the data.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Euclidean theorems are proved directly from the kernel definition, and the non-Euclidean theorems are explicitly derived as analogous expansions rather than fitted or self-citational reductions.

full rationale

The derivation chain is self-contained. The Euclidean results (exact retrieval, PL-constant ratio, effective rank, noise robustness) are proved directly from the inverse-distance kernel definition W_ij=(d(q_i,k_j)^2+epsilon)^{-1}/sum_m(...) with explicit expansions, Gershgorin bounds, and separate softmax computations; none of the target quantities is used as an input or fitted from a subset of data. The non-Euclidean theorems are transparently presented as analogues: for instance, Section 5.3 says 'Substituting these into the attention weight formula and following the same expansion as Theorem 2 yields W'_1(0)=0, W''_1(0)=-4epsilon/Delta_H^4+...'. This is a direct derivation in a new metric, not a circular reduction to the Euclidean claim or to a self-citation. The abstract itself flags the scope: 'The Euclidean theorems are proved in full; the non-Euclidean extension theorems are proved with analogous arguments.' The only self-reference, [13] (Ge, L. 2026, 'Under review'), is not used to justify any load-bearing step in the text; the Euclidean prototype is proved in Part I rather than imported from that citation. No uniqueness theorem is imported from the authors, no fitted parameter is renamed as a prediction, and no ansatz is smuggled in via citation. The paper also states its own limitations, including lack of experiments and the two-point nature of the PL analysis. A separate concern is whether the W'_1(0)=0 expansion in Theorem 2.6 is mathematically correct; that is a correctness risk, not a circularity of the derivation chain, and under the given rules it does not raise the circularity score.

Assumptions & free parameters 3 free parameters · 5 assumptions · 3 invented entities

The paper's main results depend on ε, a hand-chosen smoothing constant, on the implicit assumption that the distance kernel matrix is positive semidefinite, on the assumption that local expansion at the key captures the global PL infimum, and on the assertion that non-Euclidean analogues hold by the same arguments. No external benchmarks are used; the proof of Theorem 2.6 contains a false derivative, and the refined effective rank bound is numerically contradicted by a two-key example.

free parameters (3)
  • ε (smoothing constant) = not fit; recommended ε ≈ 0.01 * E[||q_i - k_j||^2]
    Used in every theorem (D_ij = ||q_i-k_j||^2 + ε); all PL and rank bounds depend on ε, and there is no principled value. Chosen by hand in Section 2.8.
  • w, g (FP-HIDA sparse pattern sizes) = Θ(log n)
    Set to balance complexity and coverage; not derived from data.
  • τ_base, κ, γ (DMG adaptive threshold) = tuned, values not specified
    Adaptive threshold in Lemma 9.1; regret bound depends on these hyperparameters.
assumptions (5)
  • domain assumption K=(d^2+ε)^{-1} is positive semidefinite for the distances used, so the effective rank is a meaningful eigenvalue count.
    Used in Theorem 2.8 and Lemma 5.1; for general metric spaces (Lemma 5.1) this is not proven, and for some distance matrices it can fail.
  • ad hoc to paper The global PL infimum for the 1D loss is captured by a local expansion at the exact-match point q=0.
    Theorem 2.6 proof expands around q=0 and assumes W'_1(0)=0 to get the PL constant; this premise is false, and the loss has stationary points elsewhere.
  • ad hoc to paper The Euclidean theorems extend to hyperbolic and spherical geodesic distances with only 'analogous arguments'.
    Stated in the abstract and Part II; no full non-Euclidean proofs are provided for the ten modules.
  • domain assumption Bounded query/key norms R ≤ R_max in the softmax lower bound construction.
    Theorem 2.2 assumes bounded key radius R_max to derive d = Ω((log n)^2).
  • domain assumption DMG loss is sub-Gaussian with mean μ_0 under the null model.
    Lemma 9.1 regret bound requires a sub-Gaussian assumption on the loss sequence.
invented entities (3)
  • Riemann GeoResolver framework (ten-module architecture)
    purpose: Unify hyperbolic storage and spherical routing around inverse-distance attention.
    No empirical validation; built on the flawed Euclidean base; no falsifiable handle outside the paper.
  • HyperGate
    purpose: Curvature-adaptive gating to prevent vanishing gradients.
    Gradient lower bound relies on a diagonal-dominance claim; no experiments or alternative validation.
  • Dynamic Memory Genesis (DMG)
    purpose: Online prototype allocation with O(log T) regret.
    Regret bound depends on the C-HIDA regret and sub-Gaussian assumption, both unverified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Riemann GeoResolver: A Non-Euclidean Attention Framework from Euclidean Resolver to Hyperbolic-Spherical Geometry." pith.science (2026). https://pith.science/paper/IRAAAYII

@misc{pith2026260810416,
  author       = {Pith},
  title        = {Pith review of: Riemann GeoResolver: A Non-Euclidean Attention Framework from Euclidean Resolver to Hyperbolic-Spherical Geometry},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IRAAAYII}},
  note         = {Machine review of arXiv:2608.10416}
}
abstract

We present a theoretical foundation for inverse-distance attention, from its Euclidean prototype (Resolver) to its non-Euclidean realization (Riemann GeoResolver). The Euclidean part establishes three core theorems: (1) circuit separation---IDA achieves exact retrieval with $\mathcal{O}(1)$ resources while softmax requires $\Omega((\log n)^2)$ width; (2) a Polyak--Lojasiewicz inequality with $\Omega(e^{\Delta^2/\sqrt{d}}/\Delta^2)$ stronger constant than softmax, implying linear convergence, $\mathcal{O}(\log n)$ Lipschitz scaling under a low-rank/clustering assumption, $\Theta(1)$ Hessian spread, and absence of spurious local minima; (3) a width-independent effective rank bound that limits noise memorization---softmax memorizes arbitrary labels when $d_h\ge n$, while IDA limits test error to $\mathcal{O}(\eta^2)$. The non-Euclidean extension then builds upon this prototype, replacing Euclidean distance with hyperbolic geodesic distance for storage and spherical geodesic distance for routing. The Riemann GeoResolver framework comprises ten integrated modules: four HIDA operators spanning $\Theta(n^2)$ to $\Theta(1)$ per token; Hyperbolic Curvature Compression (HCC) with provable error bounds; HyperGate with gradient lower-bound theorem; Spherical Inverse Distance Attention (SIDA) with sphere-analog PL inequalities; Dynamic Memory Genesis (DMG) with $\mathcal{O}(\log T)$ regret bounds; and Geodesic Sparse Routing (GSR) with quality and communication bounds. The Euclidean theorems are proved in full; the non-Euclidean extension theorems are proved with analogous arguments. This work establishes a theoretical arc: from Euclidean attention as a special case, to hyperbolic memory, to spherical retrieval.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 46 canonical work pages

  1. [13]

    Ge, L. (2026). GeoResolver: Optimization geometry and capacity bounds for inverse distance attention.Under review

    work page 2026

  2. [1]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS)

    work page 2017

  3. [2]

    Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. (2020). Trans- formers are RNNs: Fast autoregressive transformers with linear atten- tion. InInternational Conference on Machine Learning (ICML), PMLR 119:5156–5165

    work page 2020

  4. [3]

    Y., Ermon, S., Rudra, A., and R´ e, C

    Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and R´ e, C. (2022). FlashAt- tention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS)

    work page 2022

  5. [4]

    Dao, T. (2023). FlashAttention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691

    arXiv 2023

  6. [5]

    Child, R., Gray, S., Radford, A., and Sutskever, I. (2019). Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509

    arXiv 2019

  7. [6]

    E., and Cohan, A

    Beltagy, I., Peters, M. E., and Cohan, A. (2020). Longformer: The long- document transformer.arXiv preprint arXiv:2004.05150

    arXiv 2020

  8. [7]

    Kitaev, N., Kaiser, L., and Levskaya, A. (2020). Reformer: The effi- cient transformer. InInternational Conference on Learning Representations (ICLR)

    work page 2020

Show all 70 references
  1. [8]

    Z., Khabsa, M., Fang, H., and Ma, H

    Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. (2020). Linformer: Self-attention with linear complexity.arXiv preprint arXiv:2006.04768

    2020 arXiv

  2. [9]

    Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sar- los, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2021). Re- thinking attention with performers. InInternational Conference on Learn- ing Representations (ICLR). 32

    2021

  3. [10]

    Bello, I., Zoph, B., Vaswani, A., Shlens, J., and Le, Q. V. (2019). Attention augmented convolutional networks.arXiv preprint arXiv:1905.10863

    2019 arXiv

  4. [11]

    Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer, N., Ku, A., and Tran, D. (2018). Image transformer. InInternational Conference on Machine Learning (ICML), PMLR 80:4055–4064

    2018

  5. [12]

    McCarter, C. (2023). Inverse distance weighting attention.arXiv preprint arXiv:2310.18805

    2023 arXiv

  6. [14]

    Nadaraya, E. A. (1964). On estimating regression.Theory of Probability & Its Applications, 9(1):141–142

    1964

  7. [15]

    Watson, G. S. (1964). Smooth regression analysis.Sankhy¯ a: The Indian Journal of Statistics, Series A, 26(4):359–372

    1964

  8. [16]

    Shepard, R. N. (1968). Towards a universal law of generalization for psy- chological science.Science, 162(3850):1343–1352

    1968

  9. [17]

    Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities.Proceedings of the National Academy of Sciences, 79(8):2554–2558

    1982

  10. [18]

    and Hopfield, J

    Krotov, D. and Hopfield, J. J. (2016). Dense associative memory for pat- tern recognition. InAdvances in Neural Information Processing Systems (NeurIPS)

    2016

  11. [19]

    Ambrogioni, L. (2023). The statistical thermodynamics of generative diffu- sion models.arXiv preprint arXiv:2302.03615

    2023 arXiv

  12. [20]

    and Kiela, D

    Nickel, M. and Kiela, D. (2017). Poincar´ e embeddings for learning hierarchi- cal representations. InAdvances in Neural Information Processing Systems (NeurIPS)

    2017

  13. [21]

    and Kiela, D

    Nickel, M. and Kiela, D. (2018). Learning continuous hierarchies in the Lorentz model of hyperbolic geometry. InInternational Conference on Ma- chine Learning (ICML), PMLR 80:3779–3788

    2018

  14. [22]

    Ganea, O., B´ ecigneul, G., and Hofmann, T. (2018). Hyperbolic neural net- works. InAdvances in Neural Information Processing Systems (NeurIPS)

    2018

  15. [23]

    Chami, I., Ying, Z., R´ e, C., and Leskovec, J. (2019). Hyperbolic graph con- volutional neural networks. InAdvances in Neural Information Processing Systems (NeurIPS)

    2019

  16. [24]

    M., Battaglia, P., Bapst, V., Raposo, D., Santoro, A., and de Freitas, N

    Gulcehre, C., Denil, M., Malinowski, M., Razavi, A., Pascanu, R., Her- mann, K. M., Battaglia, P., Bapst, V., Raposo, D., Santoro, A., and de Freitas, N. (2020). Hyperbolic attention networks. InInternational Con- ference on Learning Representations (ICLR). 33

    2020

  17. [25]

    Shimizu, R., Mukuta, Y., and Harada, T. (2021). Hyperbolic neural networks++. InInternational Conference on Learning Representations (ICLR)

    2021

  18. [26]

    Chen, W., Han, X., Lin, Y., Zhao, H., Liu, Z., Li, P., Sun, M., and Zhou, J. (2021). Fully hyperbolic neural networks.arXiv preprint arXiv:2105.14686

    2021 arXiv

  19. [27]

    Liu, Q., Nickel, M., and Kiela, D. (2020). Hyperbolic graph neu- ral networks: A review of methods and applications.arXiv preprint arXiv:2002.08852

    2020 arXiv

  20. [28]

    Krioukov, D., Papadopoulos, F., Kitsak, M., Vahdat, A., and Bogu˜ n´ a, M. (2010). Hyperbolic geometry of complex networks.Physical Review E, 82(3):036106

    2010

  21. [29]

    Sala, F., De Sa, C., Gu, A., and R´ e, C. (2018). Representation tradeoffs for hyperbolic embeddings. InInternational Conference on Machine Learning (ICML), PMLR 80:4460–4469

    2018

  22. [30]

    S., Geiger, M., K¨ ohler, J., and Welling, M

    Cohen, T. S., Geiger, M., K¨ ohler, J., and Welling, M. (2018). Spherical CNNs. InInternational Conference on Learning Representations (ICLR)

    2018

  23. [31]

    Esteves, C., Allen-Blanchette, C., Makadia, A., and Daniilidis, K. (2020). Learning SO(3) equivariant representations with spherical CNNs.Interna- tional Journal of Computer Vision, 128(5):1471–1488

    2020

  24. [32]

    Bonev, B., Rietmann, M., Paris, A., Carpentieri, A., and Kurth, T. (2025). Attention on the sphere. InAdvances in Neural Information Processing Systems (NeurIPS)

    2025

  25. [33]

    J., Hirayama, J., and Kawanabe, M

    Kobler, R. J., Hirayama, J., and Kawanabe, M. (2022). Spherical con- volutions and their applications in machine learning.arXiv preprint arXiv:2201.11899

    2022 arXiv

  26. [34]

    Gu, A., Goel, K., and R´ e, C. (2021). Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396

    2021 arXiv

  27. [35]

    Gu, A., Goel, K., Gupta, A., and R´ e, C. (2022). On the parameterization and initialization of diagonal state space models. InAdvances in Neural Information Processing Systems (NeurIPS)

    2022

  28. [36]

    and Dao, T

    Gu, A. and Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752

    2023 arXiv

  29. [37]

    and Gu, A

    Dao, T. and Gu, A. (2024). Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. InInterna- tional Conference on Machine Learning (ICML)

    2024

  30. [38]

    Smith, J. T. H., Warrington, A., and Linderman, S. W. (2023). Simplified state space layers for sequence modeling. InInternational Conference on Learning Representations (ICLR). 34

    2023

  31. [39]

    Gupta, A., Gu, A., and Berant, J. (2022). Diagonal state spaces are as effective as structured state spaces.arXiv preprint arXiv:2203.14343

    2022 arXiv

  32. [40]

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. (2017). Outrageously large neural networks: The sparsely- gated mixture-of-experts layer. InInternational Conference on Learning Representations (ICLR)

    2017

  33. [41]

    Fedus, W., Zoph, B., and Shazeer, N. (2022). Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39

    2022

  34. [42]

    Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. (2021). GShard: Scaling giant models with con- ditional computation and automatic sharding. InInternational Conference on Learning Representations (ICLR)

    2021

  35. [43]

    M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A

    Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., et al. (2022). GLaM: Efficient scaling of language models with mixture-of-experts. InInternational Conference on Machine Learning (ICML), PMLR 162:5547–5569

    2022

  36. [44]

    DeepSeek-AI. (2024). DeepSeek-V3: Technical report.arXiv preprint arXiv:2412.19437

    2024 arXiv

  37. [45]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., et al. (2024). Mixtral of experts.arXiv preprint arXiv:2401.04088

    2024 arXiv

  38. [46]

    Roller, S., Sukhbaatar, S., Weston, J., et al. (2021). Hash layers for large sparse models. InAdvances in Neural Information Processing Systems (NeurIPS)

    2021

  39. [47]

    Lewis, M., Bhosale, S., Dettmers, T., Goyal, N., and Zettlemoyer, L. (2021). BASE layers: Simplifying training of large, sparse models.arXiv preprint arXiv:2103.16716

    2021 arXiv

  40. [48]

    Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., and Huang, J. (2024). A survey on mixture of experts.arXiv preprint arXiv:2407.06204

    2024 arXiv

  41. [49]

    Shazeer, N. (2019). Fast transformer decoding: One write-head is all you need.arXiv preprint arXiv:1911.02150

    2019 arXiv

  42. [50]

    Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebr´ on, F., and Sanghai, S. (2023). GQA: Training generalized multi-query transformer models from multi-head checkpoints.arXiv preprint arXiv:2305.13245

    2023 arXiv

  43. [51]

    Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S. (2024). AWQ: Activation-aware weight quantization for LLM compression and accelera- tion. InInternational Conference on Machine Learning (ICML). 35

    2024

  44. [52]

    Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. (2024). KIVI: A tuning-free asymmetric 2-bit quantization for KV cache.arXiv preprint arXiv:2402.02750

    2024 arXiv

  45. [53]

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. (2022). GPTQ: Ac- curate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323

    2022 arXiv

  46. [54]

    Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. (2024). SmoothQuant: Accurate and efficient post-training quantization for large language models. InInternational Conference on Machine Learn- ing (ICML)

    2024

  47. [55]

    W., and Keutzer, K

    Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. (2024). SqueezeLLM: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629

    2024 arXiv

  48. [56]

    Bhatnagar, P., Moradifirouzabadi, A., Yang, S.-H., Lee, S., Choi, J., and Kang, M. (2026). STAR-KV: Low-rank KV cache compression via soft thresholding for adaptive rank control. InInternational Conference on Ma- chine Learning (ICML)

    2026

  49. [57]

    Karimi, H., Nutini, J., and Schmidt, M. (2016). Linear convergence of gra- dient and proximal-gradient methods under the Polyak– Lojasiewicz condi- tion. InEuropean Conference on Machine Learning and Knowledge Discov- ery in Databases (ECML PKDD)

    2016

  50. [58]

    Belkin, M., Hsu, D., Ma, S., and Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias–variance trade-off.Proceed- ings of the National Academy of Sciences, 116(32):15849–15854

    2019

  51. [59]

    L., Kiros, J

    Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450

    2016 arXiv

  52. [60]

    He, K., Zhang, X., Ren, S., and Sun, J. (2016). Identity mappings in deep residual networks.arXiv preprint arXiv:1603.05027

    2016 arXiv

  53. [61]

    and Yuan, Y

    Li, Y. and Yuan, Y. (2018). Convergence analysis of two-layer neural net- works with ReLU activation. InAdvances in Neural Information Processing Systems (NeurIPS)

    2018

  54. [62]

    Liu, C., Zhu, L., and Belkin, M. (2022). Loss landscapes and optimization in over-parameterized non-linear systems.Journal of Machine Learning Re- search, 23(45):1–43

    2022

  55. [63]

    and Mahoney, M

    Drineas, P. and Mahoney, M. W. (2005). On the Nystr¨ om method for ap- proximating a Gram matrix for improved kernel-based learning.Journal of Machine Learning Research, 6:2153–2175. 36

    2005

  56. [64]

    He, N., et al. (2025). Hyperbolic deep learning for foundation models: A survey.arXiv preprint arXiv:2507.17787. SIGKDD 2025

    2025 arXiv

  57. [65]

    Zhang, Y., Chen, L., and Liu, Q. (2025). A survey on hyperbolic neural networks.arXiv preprint arXiv:2504.06543

    2025 arXiv

  58. [66]

    Graves, A., Wayne, G., and Danihelka, I. (2014). Neural turing machines. arXiv preprint arXiv:1410.5401

    2014 arXiv

  59. [67]

    G., Grefenstette, E., Ramalho, T., Aga- piou, J., et al

    Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska- Barwi´ nska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T., Aga- piou, J., et al. (2016). Hybrid computing using a neural network with dy- namic external memory.Nature, 538(7626):471–476

    2016

  60. [68]

    and Schmidhuber, J

    Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory.Neu- ral Computation, 9(8):1735–1780

    1997

  61. [69]

    Bengio, Y., Simard, P., and Frasconi, P. (1994). Learning long-term de- pendencies with gradient descent is difficult.IEEE Transactions on Neural Networks, 5(2):157–166

    1994

  62. [70]

    Pascanu, R., Mikolov, T., and Bengio, Y. (2013). On the difficulty of train- ing recurrent neural networks. InInternational Conference on Machine Learning (ICML), PMLR 28:1310–1318. 37

    2013

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.