{"id":"92e59de4-a288-45ad-9ed4-1c50197c97dd","arxiv_id":"2506.18943","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A spectral-dictionary VLM claims to match mid-scale baselines without convolutions or self-attention, but its O(L log L) complexity argument is not supported by the equations as written.","lead":"SDict-VLM is a 1.1-billion-parameter vision-language model that replaces convolutions and attention with learned frequency atoms. It reports competitive captioning and VQA scores, but the paper's efficiency proof relies on an FFT assumption that contradicts its own learnable frequencies.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed O(L log L) complexity is not derivable from the stated SDict operator: learned off-grid frequencies and a position-dependent phase tensor make Phi and Phi-dagger dense, so Eq. (6) omits an O(K L d) term.","rationale":"The reader's weakest assumption is the same one I would flag: the paper's advertised efficiency guarantee is unsupported by its own equations. The central claim has two parts, competitive accuracy without attention/convolution and Theta(L log L) scaling. The second part fails internally: learned off-grid frequencies and a position-dependent phase tensor make the analysis/synthesis steps dense matrix operators, so Eq. (4) and Eq. (6) do not follow from Eq. (3). Unlike the empirical issues (mixed zero-shot/fine-tuned baselines, missing error bars, absent artifacts), this is a mathematical mismatch that cannot be fixed by more careful reporting; it requires either constraining the dictionary to a uniform DFT grid and making Delta position-independent, or rewriting the complexity analysis as O(K L d). Because the headline novelty and efficiency motivation depend on this point, the REJECT verdict is appropriate. If a reviewer later verifies that a deployed implementation restricts frequencies to the grid and uses per-atom phase offsets, the efficiency claim could be revived; as written, it is not established.","tokens_in":11815,"tokens_out":11486,"duration_ms":126656,"concrete_test":"Take a trained SDict checkpoint (or simulate Eq. (3)) and record all learned f_k and Delta[n,k]. Check whether every f_k lies on the DFT grid (f_k in {0, 1/L, ..., (L-1)/L}) and every Delta[n,k] is constant in n. If either condition fails, implement Eq. (3) directly with dense complex matrix multiplications and compare per-layer GFLOPs to the two-FFT alternative at L=8192, d=1024, K=128. The direct implementation has an explicit 2 K L d term; if it is measurably larger and grows linearly in K rather than log L, the Theta(L log L) bound in Eq. (6) does not apply to the defined operator.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines SDict(X)=Re(Phi(Phi^dagger X)) with atoms phi_k[n]=A_k e^{i(2 pi f_k n + phi_k)} and explicitly states that f_k and the phase-bias tensor Delta in R^{L x K} are learned and may drift off the DFT grid. For such a dictionary, Phi^dagger=(Phi^H Phi)^{-1} Phi^H is the pseudo-inverse of a dense L x K matrix. Applying it to each of d feature channels costs Theta(K L d), and the subsequent synthesis Phi(Phi^dagger X) costs another Theta(K L d). Equations (4) and (6) instead count two radix-2 FFTs plus a Theta(K d) dense mix, omitting the factor L in the dense path. Moreover, if Delta[n,k] is position-dependent as written, the synthesis is not a DFT at all: the phase factor cannot be folded into per-bin phases, so no FFT implementation exists for the operator. The abstract and conclusion advertise Theta(L log L); that efficiency guarantee does not follow from the architecture as stated, so the central claim is not established even before the empirical comparisons are examined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes SDict-VLM, a vision-language model that replaces convolutional vision encoders and softmax self-attention with a learnable spectral dictionary mixer. It claims O(L log L) complexity and linear memory, and reports MS-COCO captioning scores (BLEU-4 39.2, CIDEr 127.5, SPICE 27.0) and VQAv2 accuracy (50.3%) for a 1.1B-parameter model, positioning it as the first VLM to eliminate both convolutions and self-attention while matching mid-scale transformer baselines. The paper includes an architectural formulation, complexity analysis, experiments against several baselines, ablations over dictionary width, and qualitative spectral visualizations.","tokens_in":11985,"tokens_out":8077,"duration_ms":77954,"significance":"If the complexity and accuracy claims held, this would be a notable contribution: a convolution-free and attention-free VLM at the 1.1B scale with competitive results on standard benchmarks, accompanied by a clean spectral-complexity narrative. The paper is clearly organized, contains explicit complexity equations, reports ablations over dictionary width, and includes a candid limitations section. However, the central complexity result depends on an FFT implementation that the operator as defined does not admit, and the benchmark comparisons mix fine-tuned and zero-shot protocols; as written, the efficiency and accuracy claims are not established.","major_comments":[{"comment":"The claimed O(L log L) complexity does not follow from the operator as defined. With learned f_k, A_k, phi_k and the position-dependent phase-bias tensor Delta in R^{L x K}, Phi in Eq. (2) is a dense L x K matrix, and Phi^dagger is its dense Moore-Penrose inverse; applying Phi^dagger and Phi to d feature channels costs Theta(K L d), not two FFTs. The phase-bias e^{i Delta[n,k]} prevents folding the phase into per-bin terms, so the operation is not a DFT and no FFT implementation exists for this operator. Additionally, the implementation sketched in Fig. 1 (FFT -> multiply by atoms -> IFFT) is a different operator from Eq. (3), yet both are used to justify Eq. (6). Thus the advertised Theta(L log L) compute and linear-memory advantages are unsupported by the paper's own equations.","section":"3.1, Eq. (4); Sec. 4, Eq. (6)"},{"comment":"The benchmark comparisons are not apples-to-apples. Table 2 explicitly marks Flamingo 9B and BLIP-2 as 0-shot, while SDict-VLM is fine-tuned on train+val; Table 1 similarly places OpenFlamingo-4B (16-shot) next to fine-tuned numbers. The abstract's claim of closing about 85% of the gap to BLIP-2 is therefore computed against zero-shot baselines, and the statement that SDict-VLM 'matches mid-scale transformer baselines' confounds architecture with training protocol. To support the claim, the authors need to compare against baselines fine-tuned under the identical data and evaluation protocol, or report SDict-VLM in the zero-shot setting alongside the zero-shot baselines.","section":"7.2, Tables 1-2; 7.1 protocol"},{"comment":"The captioning protocol appears incompatible with the stated architecture. Section 7.1 says captions are generated with nucleus sampling and length normalization, which presumes an autoregressive sequence model, yet Section 8.2 concedes that SDict's global FFT-iFFT pipeline lacks an inherent causal ordering. The paper does not describe how a noncausal global mixer is made causal during decoding, or how previously generated tokens are fed back into the model. This is not a peripheral limitation but a missing description of the inference procedure for the main benchmark, and it raises doubts about whether the reported MS-COCO captioning results can be obtained from the architecture as specified.","section":"7.1, 8.2; captioning generation"},{"comment":"No variance is reported and no artifacts are provided. The paper states that numbers are averaged over three random seeds, but Tables 1-3 give only point estimates with no standard deviations, confidence intervals, or per-seed values. The conclusion announces release of all code, checkpoints, and evaluation scripts, but the manuscript contains no URL or artifact identifier. Since the empirical contribution rests on a small number of table entries, the lack of error bars and the absence of any reproducible artifact make the results effectively unverifiable.","section":"7.1, 9; reproducibility"}],"minor_comments":[{"comment":"The sentence claiming 'outperforming PaLI-3 by a non-trivial –2.4 pp margin in BLEU-4' is self-contradictory; Table 1 shows SDict-VLM at BLEU-4 39.2 versus PaLI-3 at 40.5, so the difference is negative and the numerical value does not match the table.","section":"Section 7.2, text before Table 1"},{"comment":"The CodeCarbon estimate refers to an 'equivalently-sized ViT-GPT2 hybrid' that is not defined or listed in any table; please specify the exact baseline model, data, hardware, and whether the estimate covers pretraining or fine-tuning.","section":"Section 7.4, energy estimate"},{"comment":"The citation for GPT-4o appears as '[?]' in the text, and references [27] and [28] point to different versions of the same arXiv identifier; please resolve these reference issues.","section":"Section 2.1 and References"},{"comment":"The paper repeatedly calls the dictionary 'shared' across modalities, but Figure 1 and Section 3.2 describe vision-specific slices, text-specific slices, and a fusion dictionary initialized from the union of the two; please clarify what exactly is shared.","section":"Section 3.2 and Fig. 1"},{"comment":"Equation (4) includes an O(K^3) least-squares-solve term, but the numerical discussion after Eq. (6) only mentions Kd and treats it as negligible for L >= 1024; with K=128, K^3 is about 2.1e6 and is not negligible at L=1024, so the negligibility statement should be revisited or qualified.","section":"Section 4, discussion of Eq. (6)"}],"recommendation":"reject","confidential_remarks":"The manuscript relies heavily on the authors' own prior arXiv preprints for motivation and for the novelty claim; this is not by itself disqualifying, but the editor may want external verification that the contribution is not primarily a re-packaging of those papers. The missing GPT-4o citation and the duplicated self-reference are minor, but they add to an impression that the reference list was not carefully checked. My rejection is driven by the unsupported complexity claim and the unreconciled generative-decoding issue, not by the citation pattern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a genuinely interesting architecture idea, but the central complexity claim doesn't survive contact with the equations, and the benchmark comparisons are not like-for-like. I would not send it to review as is.\n\nWhat's new: the authors extend their prior spectral dictionary operator to multimodal data, sharing a bank of learnable frequency atoms across vision and text tokens. As far as I can tell from the cited works, this is the first VLM that tries to remove both convolutions and self-attention. The writing is clear, the ablation on dictionary width is a nice touch, and the limitations section candidly acknowledges the causal masking problem. If the architecture worked as stated, the result would matter.\n\nThe main problem is in Section 3.1 and Section 4. The atoms are complex exponentials with learned, off-grid frequencies, and there's a learned L×K phase-bias tensor Delta. For such a dictionary, Phi^dagger is a dense pseudo-inverse; applying it to each of d channels costs O(K L d), and the synthesis is also dense. The paper's O(L log L) bound assumes the analysis and synthesis are FFTs, which they are only if the frequencies lie on the DFT grid. The phase-bias tensor, because it depends on position n and atom k, can't be absorbed into per-bin phases, so there's no FFT implementation for this operator at all. This isn't a minor shortcut; it's the paper's central efficiency claim.\n\nThe empirical tables have a separate issue: Table 2 puts a fine-tuned SDict-VLM next to zero-shot baselines and uses the gap to claim competitiveness. The text acknowledges this in one sentence but still counts it as a win. There are no variance numbers despite a stated average over three seeds, no artifact link, and the references include placeholder '[?]' entries and duplicates. The submission reads like a draft.\n\nWhat's worth keeping: the shared-atom concept, the K-ablation, and the honest treatment of causality. With a corrected cost model (linear in L, but with a larger constant), matched training/evaluation protocols, and released code, this could become a solid workshop paper. As it stands, the main claim is not established.\n\nMy recommendation: desk reject, with encouragement to resubmit after fixing the math and the evaluation.","headline":"A promising architecture idea undermined by a wrong complexity analysis and mixed-protocol benchmarks; not ready for peer review.","tokens_in":12620,"tokens_out":4517,"would_cite":false,"duration_ms":42909,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A vision-language model that replaces both convolutions and attention with learned frequency atoms matches larger transformer baselines on captioning and VQA.","keywords":["spectral dictionary","vision-language model","token mixing","Fourier transform","attention-free","convolution-free","image captioning","visual question answering"],"falsifier":"Record the learned atom frequencies of a trained SDict-VLM and check whether they coincide with the discrete Fourier transform grid; if any $f_k$ differs from a multiple of $1/L$, compute the actual cost of forming $\\Phi^{\\dagger} X$ and show it scales as $O(K L d)$ rather than $O(L \\log L)$, which would refute the complexity claim while leaving the benchmark scores untouched.","tokens_in":11464,"feed_emoji":"🌊","tokens_out":8455,"duration_ms":71755,"temperature":0.7,"pith_summary":"The paper sets out to show that a spectral dictionary token mixer—representing every image patch and text token as a sparse combination of learnable frequency atoms—can replace both the convolutional vision encoder and the quadratic self-attention used for multimodal fusion. It reports a 1.1-billion-parameter prototype that reaches BLEU-4 39.2, CIDEr 127.5, and SPICE 27.0 on MS-COCO captioning and 50.3 percent accuracy on VQAv2, closing about 85 percent of the gap to a much larger BLIP-2 model. If the claims hold, the result matters because it would be the first vision-language model that is both attention-free and convolution-free at competitive accuracy, with O(L log L) scaling and linear memory that make long multimodal contexts practical.","feed_headline":"Attention-free, convolution-free VLM matches mid-scale baselines","feed_subtitle":"1.1B spectral model hits COCO CIDEr 127.5 and VQAv2 50.3% in O(L log L) time","key_machinery":"The load-bearing object is the spectral dictionary token mixer in Equation (3), an analysis-synthesis operator built from a bank of $K$ complex exponentials whose frequency, amplitude, and phase are learned per layer. The paper treats the analysis step $\\Phi^{\\dagger} X$ and the synthesis step $\\Phi S$ as FFT and inverse FFT, yielding $O(L \\log L)$ compute and $O(L)$ memory, and adds a learnable phase-bias tensor $\\Delta$ to restore locality. The dictionary is shared across image patches and wordpieces, giving cross-modal alignment in the frequency domain.","core_discovery":"The central claim is that a single learnable frequency dictionary can do the work of both spatial convolutions and softmax self-attention in a vision-language model. The paper defines the spectral dictionary operator $\\mathrm{SDict}(X)=\\mathrm{Re}(\\Phi(\\Phi^{\\dagger} X))$ with atoms $\\phi_k[n]=A_k e^{i(2\\pi f_k n+\\phi_k)}$, analyzes each token sequence into coefficients over $K=128$ atoms, mixes them in the frequency domain, and synthesizes the output; the same dictionary is shared across vision and text streams, with fusion layers using the union of both atom sets. The authors report that SDict-VLM-1.1B matches or outperforms several mid-scale transformer baselines on COCO captioning and VQAv2 while using fewer parameters, less memory, and 2.2x faster inference than one of them, and claim this is the first VLM to remove both convolutions and self-attention.","pith_inferences":["The paper leaves implicit that its $O(L \\log L)$ bound depends on the learned frequencies $f_k$ remaining on the uniform DFT grid; if they drift, the analysis step $\\Phi^{\\dagger} X$ is no longer a Fourier transform and would cost $O(K L d)$ to evaluate. A direct complexity test with recorded atom frequencies would settle this.","A natural testable extension is to turn the fixed dictionary width $K$ into a data-adaptive budget, allocating atoms to high-entropy spectral bands and pruning unused ones; the paper outlines this idea but does not test it.","The absence of causal masking in the global FFT suggests the design is better suited to bidirectional captioning and VQA than to autoregressive or streaming generation; a sliding-window or dilated spectral basis would be needed to extend it, and the paper lists this as future work.","Spectral dictionaries could be compared against state-space models as alternative attention replacements; the paper mentions this possibility but does not test it."],"forward_implications":["If the central claim is correct, vision-language models no longer need convolutions or self-attention; a single spectral mixer can carry both modalities.","The $O(L \\log L)$ scaling and linear activation memory imply that contexts of tens of thousands of tokens, such as thousands of image patches plus text, fit on the same GPU budget that would require activation checkpointing for attention.","The shared frequency dictionary provides a built-in interpretability route: atom spectra can be inspected to see which frequency bands drive particular answers or caption phrases.","Because dictionary width $K$ is independent of sequence length, the same pretrained atoms can be reused across input resolutions without adding weights.","The accuracy-compute trade-off is tunable: reducing $K$ from 128 to 64 cuts FLOPs and lowers CIDEr by 7.9 points, while increasing $K$ beyond 128 gives almost no gain."],"supporting_citations":[{"why":"Supplies the Fourier-mixing baseline in language that the paper extends from text-only to multimodal.","marker":"[1]"},{"why":"Shows learnable global filters in the frequency domain match transformer accuracy on images.","marker":"[2]"},{"why":"Introduces the dictionary-learning idea with learnable frequency, amplitude, and phase for images.","marker":"[3]"},{"why":"Shows the same dictionary concept works for language modeling at reduced FLOPs.","marker":"[4]"},{"why":"The large captioning and VQA baseline whose performance gap the paper measures at about 85 percent closed.","marker":"[7]"},{"why":"The mid-scale transformer baseline used for parameter, memory, and speed comparisons.","marker":"[8]"}],"fun_headline_variants":["No attention, no conv: spectral VLM matches mid-scale baselines","Spectral dictionary VLM: no attention, no conv, matches mid-scale baselines","1.1B spectral VLM: no attention, no conv, 85% of BLIP-2 gap closed","Frequency-domain mixer replaces attention and conv in VLM, matches baselines","First VLM without attention or conv, matches mid-scale baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The compute advantage in the paper rests on treating the learned frequency atoms as if they always form a uniform Fourier grid; if the learned frequencies drift off that grid, the analysis step is not a fast Fourier transform and the advertised O(L log L) cost is not established.","fun_headline_variants_meta":{"raw":{"variants":["No attention, no conv: spectral VLM matches mid-scale baselines","Spectral dictionary VLM: no attention, no conv, matches mid-scale baselines","1.1B spectral VLM: no attention, no conv, 85% of BLIP-2 gap closed","Frequency-domain mixer replaces attention and conv in VLM, matches baselines","First VLM without attention or conv, matches mid-scale baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001105,"raw_usage":{"total_tokens":4634,"prompt_tokens":996,"completion_tokens":3638,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":3530}},"tokens_in":612,"tokens_out":3638,"duration_ms":22967,"temperature":1.0,"reasoning_tokens":3530,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:53:40.333139+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record the learned atom frequencies of a trained SDict-VLM and check whether they coincide with the discrete Fourier transform grid; if any $f_k$ differs from a multiple of $1/L$, compute the actual cost of forming $\\Phi^{\\dagger} X$ and show it scales as $O(K L d)$ rather than $O(L \\log L)$, which would refute the complexity claim while leaving the benchmark scores untouched.","supporting_citations":[{"cited_title":"FNet: Mixing Tokens with Fourier Transforms,","cited_arxiv_id":null,"evidence_quote":"Supplies the Fourier-mixing baseline in language that the paper extends from text-only to multimodal."},{"cited_title":"Global Filter Networks for Image Classification,","cited_arxiv_id":null,"evidence_quote":"Shows learnable global filters in the frequency domain match transformer accuracy on images."},{"cited_title":"Spectral Dictionary Learning for Generative Image Modeling","cited_arxiv_id":"2504.17804","evidence_quote":"Introduces the dictionary-learning idea with learnable frequency, amplitude, and phase for images."},{"cited_title":"BLIP-2: Bootstrapping Language-Image Pre-training,","cited_arxiv_id":null,"evidence_quote":"The large captioning and VQA baseline whose performance gap the paper measures at about 85 percent closed."}],"review_version":1}