{"id":"0d303495-7708-4e70-833e-271c2dc9d2f9","arxiv_id":"2501.15757","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CKANs are measurably less efficient than standard CNNs, and on ImageNet the accuracy gap is large, but the paper's baseline and timing comparisons are not controlled.","lead":"This paper trains convolutional Kolmogorov-Arnold networks on ImageNet, MNIST, and a drug-response tabular dataset, and compares their speed, size, and accuracy with standard CNNs. It reports that CKANs roughly match CNNs on small datasets but are far slower and less accurate at ImageNet scale, though the comparison uses a pretrained AlexNet baseline and a historical training-time estimate.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central ImageNet accuracy claim compares a from-scratch CKAN against a pretrained AlexNet checkpoint, so the reported 14–16 pp gap may largely reflect training-recipe differences rather than architectural limitations.","rationale":"The reader's weakest_assumption identifies the unfair baseline as the key soft spot, and I agree. The strongest claim — that CKANs are not competitive at ImageNet scale — depends directly on the 42.8% versus 56.6% comparison in Section IV-D. Because the AlexNet baseline is a pretrained PyTorch checkpoint rather than a from-scratch model trained under the same recipe as AlexNet-KAN, the accuracy gap conflates architecture with training setup. This is load-bearing: if a properly matched AlexNet underperforms the pretrained checkpoint, the quantitative headline weakens, even if the qualitative direction (CKAN slower and less accurate) may survive. The paper's own limitation note about lack of low-level spline accelerators partially addresses the efficiency gap, but not the unfair baseline. The 48-day versus 'at most 3 days' training-time comparison is also questionable since it relies on a hand calculation from the original AlexNet report rather than a measured run, but the accuracy comparison is more central to the paper's main contribution. The proposed concrete test — training AlexNet from scratch under the identical protocol — would settle whether the claimed accuracy gap is genuine. No ad hominem is intended; the issue is experimental design, not author integrity.","tokens_in":14497,"tokens_out":3528,"duration_ms":32923,"concrete_test":"Train a standard AlexNet from scratch with the exact AlexNet-KAN protocol: ImageNet preprocessing limited to normalization, batch size 16, 4x K80 GPUs with PyTorch DDP, Adam optimizer with the same learning rate, 100 epochs, early stopping patience 3, and no pretrained weights or data augmentation beyond normalization. Compare final validation Top-1/Top-5 accuracy and wall-clock training time on the same cluster. If the from-scratch AlexNet Top-1 accuracy falls close to 42.8%, the claimed 14–16 pp gap is an artifact of comparing against a pretrained checkpoint; if it remains near 56%, the central claim holds as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-A2 states, 'We use PyTorch's pretrained AlexNet for CNN's part as it is the industry standard.' The central quantitative claim in Section IV-D — CKAN at 42.8% / 67.8% Top-1 / Top-5 versus AlexNet at 56.6% / 79.1% — therefore does not control for training protocol. The CKAN model was trained from scratch using only normalization, a batch size of 16, 4x K80 DDP, and 100 epochs with early stopping; the pretrained AlexNet was trained with modern augmentations, a tuned schedule, and different hardware. A from-scratch AlexNet trained under the CKAN's exact recipe would likely achieve lower accuracy than 56.6%, shrinking the headline gap. The paper also provides no error bars or seed repetitions for this ImageNet comparison, so a single run cannot establish the claimed 14–16 pp deficit as an architectural property. If the from-scratch AlexNet accuracy lands near 42.8%, the central claim that 'CKANs do not scale gracefully' would need substantial qualification.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical comparison of Convolutional Kolmogorov-Arnold Networks (CKANs) against standard CNN baselines on three tasks: ImageNet-1k with an AlexNet-style CKAN, MNIST with a LeNet-style CKAN, and the MoA tabular dataset with a 1D CNN. The authors report accuracy, precision, recall, F1, FLOPs, inference time, training time, and parameter counts, and add a 24-run ablation on LeNet/MNIST over spline grid size, width multiplier, ReLU use, and pruning ratio. The main conclusion is that CKANs are competitive on small datasets but fail to scale to ImageNet, attaining 42.8% Top-1 accuracy versus 56.6% for AlexNet while taking longer to train and infer.","tokens_in":14727,"tokens_out":5610,"duration_ms":48822,"significance":"If the comparisons were properly controlled, the paper would provide a useful negative result on CKAN scalability and a rare large-scale CKAN training data point. The ablation study is systematic, the code is released, and the authors are transparent about the lack of low-level spline accelerators. However, the central ImageNet claim rests on an uncontrolled baseline: the CNN accuracy comes from a pretrained PyTorch AlexNet, and the CNN training time is a historical hand estimate, not a same-protocol measurement. As a result, the headline 14-16 percentage-point gap cannot currently be attributed to the CKAN architecture.","major_comments":[{"comment":"The primary evidence for the paper's central claim is the comparison between CKAN and AlexNet on ImageNet. Section III-A2 states that 'We use PyTorch's pretrained AlexNet for CNN's part as it is the industry standard,' which means the 56.6% / 79.1% Top-1 / Top-5 numbers are not produced under the same training protocol as the CKAN model. The pretrained checkpoint was trained with different augmentations, schedule, optimizer settings, batch size, and hardware. A from-scratch AlexNet trained with the CKAN's recipe (100 epochs, batch size 16, 4xK80, normalization only) would likely score lower, potentially narrowing the gap substantially. To support the claim that CKANs 'do not scale gracefully,' the authors must train the CNN baseline from scratch under the identical data pipeline, optimizer, epoch budget, and hardware, and report those numbers.","section":"III-A2, IV-D, Table III"},{"comment":"The claim of a 16x training-time increase ('48 days' vs 'at most 3 days') is not a measured comparison. Section IV-A describes the AlexNet baseline time as 'by hand calculation of the original report,' i.e., a historical estimate from Krizhevsky et al. on 2012-era hardware, whereas the CKAN run was measured on 4x Tesla K80 GPUs with a modern software stack. Different hardware, libraries, and data pipelines make this ratio meaningless as an efficiency comparison. The authors should either measure both models on the same cluster or clearly label the historical number as context and exclude it from the efficiency conclusions.","section":"IV-A"},{"comment":"The headline ImageNet results are single runs with no error bars or seed repetitions, so the 14-16 pp gap cannot be separated from run-to-run and training-recipe variance. Additionally, the comparison is not parameter-matched: AlexNet KAN has 39,756,776 parameters versus 61,100,840 for AlexNet, so the accuracy gap may partly reflect capacity differences. At minimum, the authors should report mean and standard deviation over multiple seeds and include a matched-parameter or matched-capacity variant of the baseline before attributing the gap to the spline-based representation.","section":"Table III, IV-D"}],"minor_comments":[{"comment":"The FLOPS cell for Tabular CKAN is malformed: '798,61,586' appears to be '79,861,586'; please correct and verify all numeric formatting.","section":"Table III"},{"comment":"The text reports LeNet-KAN F1 as 97.79 while Table III reports 98.79; these numbers should agree.","section":"IV-C"},{"comment":"The paper uses 'FLOPS' interchangeably for a count of operations; FLOPs (count) and FLOPS (rate) should be distinguished.","section":"Throughout"},{"comment":"Equation (7) has index errors: the feature-map formula uses 'a_{i-x,i-y}' but the kernel indices are inconsistent with Equations (8)-(9); please rewrite the convolution definition cleanly.","section":"Eq. (7)-(10)"},{"comment":"The MNIST-based conclusion that 'we therefore set an upper bound of g=8 for any CKAN deployed in resource-constrained environments' is overgeneralized; please qualify this recommendation to the tested setting.","section":"V.B"},{"comment":"Section VI-D mentions 'ImageNet-100 pilot runs' that are not described anywhere in the results; either document the pilot or remove the reference.","section":"VI-D"},{"comment":"The precision, recall, and F1 definitions for the multi-class ImageNet task do not specify whether macro, micro, or weighted averaging is used; please state the averaging scheme.","section":"III-C, Table III"}],"recommendation":"major_revision","confidential_remarks":"Given that the main quantitative claim rests on a pretrained baseline, I would not accept the paper as is. If the authors can retrain AlexNet under the same protocol and add seed variance, the result would be worth revisiting; otherwise, the ImageNet section should be reframed as an uncontrolled case study rather than a head-to-head benchmark. The manuscript may also be short of the expected standard of a full ImageNet benchmark because it uses only AlexNet and a small batch size."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the first ImageNet-1k training run of convolutional KANs (per the paper's own Table I) plus a well-structured 24-configuration ablation on LeNet. That is real work, and the qualitative conclusion — CKANs are compute-hungry and do not beat CNNs at scale — is probably right. The ablation is carefully documented with a design matrix, metrics, and code, and the pruning analysis is a useful practical contribution. The paper also states its own limitations honestly, including the lack of low-level spline accelerators.\n\nThe soft spots are real and they hit the central quantitative claims. The AlexNet baseline is PyTorch's pretrained checkpoint, not a model trained under the same protocol as the CKAN; Section III-A2 says so explicitly. So the reported 42.8% vs 56.6% Top-1 gap (14–16 pp) may substantially reflect training-recipe differences rather than architecture. The 16x training-time gap is a hand estimate from the original 2012 AlexNet paper, not a measured run on the same hardware, and no error bars or seed repeats are reported for the main ImageNet comparison. Those are not minor footnotes; they are the paper's central evidence. The ablation on MNIST is solid, but the paper extrapolates from it to recommend a default variant for tabular tasks without testing that variant on the tabular dataset.\n\nOn the citation pattern: no self-citations, and the related-work table is genuinely useful. The paper is clearly written and the authors engage with prior KAN skepticism (Yu et al.) rather than ignoring it.\n\nMy bottom line: the direction of the finding is likely correct, but the quantitative strength claimed in the abstract and Section IV is not supported by the experiments as run. A controlled from-scratch AlexNet baseline under the same training recipe and measured timings would make this a credible negative result. As is, I would not cite the headline numbers, but I would point people to the ablation and the first ImageNet run.\n\nThis deserves peer review, not desk rejection. It is a useful benchmark and the flaws are fixable with a major revision. I'd recommend the editor send it out, with a referee instructed to focus on the baseline fairness and timing methodology.","headline":"First ImageNet-scale CKAN benchmark with a clean ablation, but the headline accuracy and timing numbers compare against a pretrained baseline and an estimated training time, so the central quantitative claim is not yet supported.","tokens_in":15261,"tokens_out":1320,"would_cite":false,"duration_ms":13689,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that Convolutional Kolmogorov-Arnold Networks match CNNs on small datasets but fail on ImageNet-1k, with 42.8% Top-1 accuracy versus 56.6% for AlexNet at double the FLOPs and four times the latency.","keywords":["Convolutional Kolmogorov-Arnold Networks","KAN","ImageNet-1k","AlexNet","LeNet","tabular classification","B-spline","efficiency bottleneck"],"falsifier":"Train a standard AlexNet from scratch on ImageNet-1k for 100 epochs with the same batch size, optimizer, learning-rate schedule, data pipeline, and GPU hardware used for AlexNet-KAN, and compare Top-1 accuracy and wall-clock training time; if the gap narrows to a few points or the CKAN's 48-day figure is no longer compared with a comparable 100-epoch CNN run, the central efficiency claim would need revision.","tokens_in":1747,"feed_emoji":"⚡","tokens_out":2915,"duration_ms":84909,"temperature":0.7,"pith_summary":"This paper tests whether Convolutional Kolmogorov-Arnold Networks (CKANs), which replace learned convolutional kernel weights with trainable B-spline functions, can serve as practical replacements for standard CNNs. On the small-scale MNIST benchmark, a CKAN LeNet matches a CNN's accuracy within 0.1 percentage point but at roughly seven times the FLOPs and more than four times the inference latency. On ImageNet-1k the gap becomes decisive: the authors report 42.8% versus 56.6% Top-1 accuracy for AlexNet-KAN versus AlexNet, with double the FLOPs and four times the latency. On a tabular MoA dataset, the CKAN variant also trails the matched CNN on accuracy, recall, and F1 while training 1.65 times longer. The paper concludes that CKANs are currently compute-heavy specialists best suited to small scientific or tabular tasks, not large-scale vision.","feed_headline":"KAN-based CNNs fall 14 points behind AlexNet on ImageNet","feed_subtitle":"They also burn about twice the FLOPs and four times the inference latency, undercutting their appeal.","key_machinery":"The central object is the CKAN convolutional layer, which replaces each entry of a CNN kernel with a trainable B-spline function, so the feature map is computed by evaluating splines on pixel windows rather than by multiplying fixed weights. The parameter count grows roughly as the product of input and output dimensions times the number of spline intervals plus polynomial order, making KAN layers about four times heavier per channel than ordinary convolutions; the paper compensates by reducing filter counts in the CKAN variants. The argument runs through three matched model pairs (LeNet, AlexNet, and a 1-D tabular CNN), plus a 24-run ablation sweep on LeNet that varies spline grid size, channel width, ReLU on/off, and 25% structured pruning to isolate which design choice drives the cost.","core_discovery":"The paper's central claim is that CKANs are not competitive at ImageNet scale: the spline-based convolutional layer does not capture the hierarchical abstractions required for large-scale visual recognition. In their head-to-head comparison, AlexNet-KAN attains 42.8% Top-1 and 67.8% Top-5 accuracy, while standard AlexNet reaches 56.6% and 79.1%, even though the KAN variant uses roughly double the FLOPs and four times the inference latency. On MNIST the same pattern holds in miniature: accuracy parity within 0.1 percentage point, but a seven-fold FLOP increase and 3.28x per-image latency. The authors attribute the bottleneck to the B-spline activation functions and the absence of low-level spline accelerators, and their 24-run ablation on LeNet identifies spline grid size as the primary cost lever: increasing knots from 4 to 8 yields only +0.19 percentage points while raising FLOPs by 77%.","pith_inferences":["Inference: a fairer protocol would compare CKANs against a modern lightweight CNN trained and tuned under the same recipe, which would test whether the gap is specific to spline kernels or to the dated AlexNet backbone.","Inference: the reported seven-fold FLOP increase on LeNet suggests that even at small scale CKANs' practical cost is in spline evaluation and memory bandwidth, not parameter count; replacing B-splines with faster basis functions or lookup tables is a testable lever the paper mentions only in future work.","Inference: the authors' suggestion that CKANs are better suited to tabular and scientific tasks is not fully established by their MoA result, since the CKAN still lost on accuracy, recall, and F1; additional benchmarks would be needed to support that positioning.","Inference: a direct test of the Kolmogorov-Arnold representation theorem's promise, namely interpretability gains from univariate functions, is absent from this paper; whether CKANs offer any scientific interpretability advantage on the MoA dataset remains open."],"forward_implications":["If CKANs cannot beat a 2012 CNN on ImageNet at higher cost, they are not a practical replacement for deep CNNs in large-scale computer vision without further architectural or kernel-level advances.","The ablation's Fast-KAN-Lite configuration (4 spline grid, 1.5x width, ReLU on, 25% structured pruning) gives the best accuracy-efficiency trade-off on MNIST and is the recommended default for small-scale tasks.","Structured 25% pruning is necessary to make CKANs competitive; it cuts latency by roughly 35% and parameters by 13%, but pruned CKANs remain 1.5-2.0x slower than depth-matched CNNs.","Spline grid resolution is the dominant cost lever: grids above 8 knots degrade accuracy while roughly doubling latency, so deployment should keep the grid at 8 or below.","The paper reports that its convolutional kernels were implemented without low-level spline accelerators and estimates that kernel-level optimization, quantization, or curriculum grid growth could cut CKAN latency by 30-50%."],"supporting_citations":[{"why":"Introduces the KAN architecture and the B-spline activations that CKAN layers replace CNN kernels with.","marker":"[8]"},{"why":"Defines the Convolutional KAN layer that is the main object under scrutiny.","marker":"[14]"},{"why":"Defines AlexNet, the ImageNet baseline, and the historical training-time figure used in the comparison.","marker":"[15]"},{"why":"Provides the 1-D tabular CNN architecture and weighted loss replicated for the MoA comparison.","marker":"[16]"},{"why":"Supplies the ImageNet-1k dataset with 1.3 million images.","marker":"[17]"},{"why":"Supplies the MNIST dataset used for the LeNet comparison and ablation sweeps.","marker":"[18]"},{"why":"Supplies the MoA dataset and the BCE evaluation protocol for the tabular comparison.","marker":"[19]"},{"why":"Establishes the parameter-count formulas and prior evidence that KANs lag MLPs in vision tasks.","marker":"[23]"},{"why":"Supplies the structured pruning method used in the ablation study.","marker":"[44]"}],"fun_headline_variants":["KAN convolutions: 14-point ImageNet drop, 4x latency","Convolutional KANs lose to CNNs on ImageNet scale","Spline layers make KAN-CNNs slow and weak on ImageNet","CKANs: 7x FLOPs on MNIST, 4x latency on ImageNet","KAN-based CNNs struggle where datasets get big"],"cache_read_input_tokens":17408,"weakest_assumption_plain":"The headline comparison assumes that using a pretrained AlexNet, together with the historical 'at most 3 days' training-time figure, is a fair proxy for a CNN trained from scratch under the same protocol as the CKAN; if AlexNet were retrained with identical epochs, hardware, and data pipeline, the accuracy and training-time gaps could shrink substantially.","fun_headline_variants_meta":{"raw":{"variants":["KAN convolutions: 14-point ImageNet drop, 4x latency","Convolutional KANs lose to CNNs on ImageNet scale","Spline layers make KAN-CNNs slow and weak on ImageNet","CKANs: 7x FLOPs on MNIST, 4x latency on ImageNet","KAN-based CNNs struggle where datasets get big"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000338,"raw_usage":{"total_tokens":1905,"prompt_tokens":1019,"completion_tokens":886,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":784}},"tokens_in":635,"tokens_out":886,"duration_ms":7301,"temperature":1.0,"reasoning_tokens":784,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:57:42.570294+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a standard AlexNet from scratch on ImageNet-1k for 100 epochs with the same batch size, optimizer, learning-rate schedule, data pipeline, and GPU hardware used for AlexNet-KAN, and compare Top-1 accuracy and wall-clock training time; if the gap narrows to a few points or the CKAN's 48-day figure is no longer compared with a comparable 100-epoch CNN run, the central efficiency claim would need revision.","supporting_citations":[{"cited_title":"Learning both weights and con- nections for efficient neural network,","cited_arxiv_id":null,"evidence_quote":"Supplies the structured pruning method used in the ablation study."},{"cited_title":"Kaggle MoA 2nd place solution,","cited_arxiv_id":null,"evidence_quote":"Provides the 1-D tabular CNN architecture and weighted loss replicated for the MoA comparison."},{"cited_title":"The MNIST database of handwritten digits,","cited_arxiv_id":null,"evidence_quote":"Supplies the MNIST dataset used for the LeNet comparison and ablation sweeps."},{"cited_title":"Mechanisms of action (MoA) prediction,","cited_arxiv_id":null,"evidence_quote":"Supplies the MoA dataset and the BCE evaluation protocol for the tabular comparison."}],"review_version":1}