{"id":"bdce21f4-c1e0-4c19-bceb-b27325b59004","arxiv_id":"2505.09366","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"KAN and FKAN did not significantly outperform MLP and CNN for turn intent prediction, and user-specific training only helped conventional models, not deep learning models.","lead":"This paper tests whether Kolmogorov-Arnold Networks with learnable activations beat standard networks for detecting turns in a lower-limb prosthesis, using IMU data from five amputees. It finds they do not, and that deep learning models can be trained on data from multiple users without losing performance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-participant Wilcoxon tests on ten overlapping test divisions violate independence; corrected tests are needed before claiming per-participant significance for ML models.","rationale":"The paper's central claims include a positive result (user-specific training superior to pooled for ML models, p < 0.05) and a null result (no significant difference for DL models). The reader's weakest assumption targets the independence violation in the per-participant Wilcoxon tests. I agree this is a real flaw, but it may not fully undermine the abstract's aggregate p-value because the Bayesian paired t-test uses one average per participant. However, the paper explicitly uses the per-participant Wilcoxon results to claim significance 'for each participant,' and those results are unreliable. The more significant inferential gap is the DL equivalence overreach: the paper interprets non-significance as 'comparable performance,' which is not statistically justified without an equivalence margin or adequate power. Given the small sample (n=5) and the overlapping-window design, both issues support the reader's CONDITIONAL verdict rather than moving it. The proposed tests would directly address both the independence problem and the equivalence claim.","tokens_in":14221,"tokens_out":13253,"duration_ms":129966,"concrete_test":"Recompute the per-participant Hypothesis 2 comparisons using a permutation test that respects the trial structure: randomly permute the specific/pooled condition within each participant while preserving temporal dependence (e.g., block-permute whole trials), and compare the observed macro-F1 differences to the permutation distribution. Also run a two-one-sided t-test (TOST) with a pre-specified margin (e.g., ±2 macro-F1 points) for the CNN and FKAN specific-vs-pooled differences to test whether the data actually establish equivalence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The per-participant Wilcoxon signed-rank tests in Section II-D and Figs. 4-5 compare model performance across ten stratified random divisions of each participant's test set. These divisions are not independent paired samples: the underlying windows are extracted with 50% overlap (Section II-A5), so windows in different divisions share data samples, and even non-overlapping divisions contain autocorrelated time-series observations from the same trials. Treating the ten F1 scores as independent paired observations underestimates the variance of the test statistic and inflates the significance of the reported p-values. This directly affects the statement that 'The Wilcoxon test indicated a statistically significant difference for MLP and KAN for specific vs pooled training in each participant (p < 0.01 and p = 0.02)' and the per-participant p-values in Fig. 5. The abstract's aggregate p < 0.05 for ML models may rely on the Bayesian paired t-test across participants, which uses one value per participant and is less affected by within-participant division overlap; however, the paper also uses the invalid Wilcoxon results as per-participant evidence. Additionally, the non-significant DL comparison is interpreted as evidence that pooled training is 'comparable' to user-specific training, but equivalence cannot be concluded from a null result with only five participants; an equivalence test or a defined margin is needed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript evaluates Kolmogorov-Arnold Networks (KAN) and fractional Kolmogorov-Arnold Networks (FKAN) for turn-intent prediction in a variable-stiffness torsion adapter for lower-limb prostheses, using IMU-derived data from five individuals with amputation. The authors compare KAN against MLP and FKAN against CNN to test whether learnable activation functions improve macro-averaged F1 over static activation functions, and they compare user-specific versus pooled training. Their reported results are that learnable activations do not significantly improve performance, that user-specific training outperforms pooled training for MLP and KAN (aggregate p < 0.05), and that no significant difference is observed for CNN and FKAN, which they interpret as pooled training being comparable for deep learning models.","tokens_in":14490,"tokens_out":5798,"duration_ms":55249,"significance":"If the claims hold, the paper provides one of the first assessments of KAN and FKAN for lower-limb prosthetic control, with a carefully constructed experimental setup that includes Bayesian hyperparameter optimization, class weighting, macro-F1 evaluation, and comparisons against a prior SVM baseline and Auto-sklearn. The per-participant analyses and the observation that KAN inference is substantially slower than MLP are practically useful for real-time control constraints. However, the statistical analysis of per-participant comparisons is flawed, and the equivalence claim for deep learning models is not supported by the data, so the reported significance of the user-specific versus pooled findings is overstated as presented.","major_comments":[{"comment":"The per-participant Wilcoxon signed-rank tests compare ten stratified random divisions of each participant's test set. Because these divisions are overlapping subsamples of the same test data, and because the underlying sliding windows themselves overlap by 50% (Section II-A5), the observations are not independent. Treating the ten F1 scores as independent paired samples violates the independence assumption of the Wilcoxon test and inflates the significance of the per-participant p-values reported in Fig. 5 (for example, 'p < 0.01' for MLP and 'p = 0.02' for KAN for each participant). This invalidates the per-participant evidence for Hypothesis 2 and similarly affects the per-participant Hypothesis 1 comparisons in Fig. 4. The authors should either use a single test-set evaluation per participant and test across participants, or use a dependence-aware resampling scheme such as a block bootstrap stratified by trial. The aggregate Bayesian paired t-test in Table IV uses one average per participant and is less affected by this issue, but the per-participant p-values in Figs. 4-5 should be corrected or removed.","section":"Section II-D, Figs. 4-5"},{"comment":"The manuscript concludes that pooled training 'showed comparable performance' for deep learning models and that 'deep learning achieves similar performance using pooled data,' based on the absence of a statistically significant difference (Table IV: CNN p = 0.127, FKAN p = 0.833). With only five participants, a non-significant result does not establish equivalence. The authors should either conduct an equivalence test with a pre-specified margin (for example, two one-sided tests) or limit the conclusion to 'no evidence of a difference' and remove the strong claim of comparable performance.","section":"Abstract, Section IV, Section V"},{"comment":"The relationship between the test set construction described in Section II-A4 (one trial from each turn type at each stiffness setting, yielding nine turning events and three straight-walking trials) and the 'ten divisions of test data' used for the statistical tests is unclear. It is not evident whether the ten divisions are independently drawn subsets, overlapping bootstrap samples, or something else. Please define the divisions explicitly, and state whether they are overlapping or disjoint, since this is directly relevant to the validity of the Wilcoxon tests.","section":"Section II-A4 and II-D"}],"minor_comments":[{"comment":"There is a typo in 'specifc vs pooled training' that should read 'specific vs pooled training.'","section":"Section III"},{"comment":"The text states that KAN inference is 'approximately 1000 times greater' than MLP, but Table VIII shows total inference times of 0.2 s versus 0.02 s for participant A01, which is a factor of 10. Please reconcile the numbers or clarify the basis for the 1000-fold claim.","section":"Section IV and Table VIII"},{"comment":"The 'Bayesian paired t-test' results are reported as p-values (for example, 0.965, 0.002). A Bayesian analysis typically yields a posterior probability or a Bayes factor rather than a frequentist p-value; please specify the prior, the test statistic, and the decision rule used.","section":"Section II-D and Table IV"},{"comment":"The phrase 'parallel processing, which may reduce inference speed for prediction' is ambiguous; parallel processing generally reduces latency. Please rephrase to state the intended direction of the effect.","section":"Introduction, Section I"},{"comment":"The violin plots aggregate the ten overlapping test divisions; given the dependence concern, it might be clearer to display the individual per-division scores as points or to show a single test-set evaluation per participant.","section":"Figs. 4-5"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical benchmark that could be useful for the prosthetic control community, but the statistical reporting needs substantial revision. The per-participant Wilcoxon tests are invalid for the stated design, and the equivalence claim for deep learning models is an overreach from a null result with five participants. The central claim that user-specific training benefits conventional ML models may still survive if the aggregate Bayesian test is properly used, but the authors need to correct the analysis and moderate the conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this is the first test of KAN and FKAN for lower-limb prosthesis turn intent, and it reports a clean negative result — learnable activations do not beat MLP or CNN on this three-class problem. That is useful and the paper is honest about it.\n\nWhat it does well: the experiment design is careful. Bayesian hyperparameter optimization, class weighting, macro F1, comparison against Auto-sklearn and the previous SVM baseline, and a user-specific vs. pooled training comparison that extends the earlier work. The FKAN architecture comes from a co-author, but the result doesn't favor it, so self-citation isn't a problem here. The data and code availability aren't clear, but the description is sufficiently detailed for replication.\n\nThe soft spots are statistical. The per-participant Wilcoxon signed-rank tests treat ten stratified random divisions of the same test set as independent samples. Those divisions share overlapping windows (50% overlap) and autocorrelated time series, so the independence assumption fails. The p-values for Hypothesis 2 per subject (Fig. 5) are therefore anti-conservative. The abstract's aggregate p<0.05 for ML models likely rests on the Bayesian paired t-test across five participants, which is less affected because each participant contributes one average — but the paper also leans on the invalid Wilcoxon results in the text. Fixing this means either using a proper paired test with one value per participant or a mixed model, or doing bootstrap with respect to subjects. Also, the claim that pooled training is 'comparable' to user-specific for DL models is a null result with only five people; equivalence testing or a defined margin is needed before drawing that conclusion.\n\nThe central negative result, however, is robust. Even with softer statistics, the KAN and FKAN F1 scores are not meaningfully better than MLP/CNN, and the inference time of KAN is orders of magnitude higher — that's a practical knockout for this application.\n\nThis paper is for prosthesis control engineers and researchers tracking KAN's real-world utility. It deserves a serious referee: the topic is timely, the experiment is honest, and the statistical flaws are fixable. I'd send it to review with a request to reanalyze the per-participant comparisons and soften the equivalence claim.","headline":"A first, honest KAN benchmark for prosthesis turn intent that finds no advantage over static activations, but its per-participant significance claims rely on non-independent test divisions.","tokens_in":14977,"tokens_out":2214,"would_cite":false,"duration_ms":20363,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports that KAN and FKAN, with their learnable activation functions, did not significantly beat MLP and CNN for lower-limb prosthetic turn prediction, and that deep models performed as well trained on pooled multi-participant…","keywords":["Kolmogorov-Arnold networks","prosthesis control","turn intent prediction","inertial measurement unit","learnable activation functions","deep learning","user-specific training","pooled training"],"falsifier":"Re-evaluate the specific-versus-pooled comparisons on a single held-out test set per participant, or with block bootstrap over non-overlapping windows, so each observation is independent; if the MLP/KAN advantage over pooled training ceases to be significant at p < 0.05, the paper's Hypothesis 2 support would not survive.","tokens_in":14067,"feed_emoji":"🦿","tokens_out":11543,"duration_ms":102241,"temperature":0.7,"pith_summary":"This paper asks whether Kolmogorov-Arnold Networks (KANs), which replace fixed neuron activations with learnable functions, improve turn-intent prediction for a lower-limb prosthesis controller, and whether training on a single user's data beats training on pooled data from several users. Using simulated shank IMU data from five people with transtibial (below-knee) amputations, it compares KAN against a multilayer perceptron (MLP) and a fractional KAN (FKAN) against a convolutional neural network (CNN), scoring macro-averaged F1 on three classes: straight walking, pre-turn swing, and turning stance. The main result is a null result: learnable activations were not significantly better than static activations, and for the deep models no significant difference appeared between user-specific and pooled training. For the conventional models, user-specific training was significantly better. The practical stake is whether a prosthesis controller can be trained once on pooled data or must be personalized, and whether the extra complexity of learnable activations buys anything.","feed_headline":"KAN learnable activations don't beat MLP/CNN for turn control","feed_subtitle":"User-specific training still matters for shallow models, but deep models can share data across prosthesis users.","key_machinery":"The object carrying the comparison is the Kolmogorov-Arnold Network, a neural architecture that puts learnable univariate functions—B-spline bases in KAN, fractional Jacobi basis functions in FKAN—on the edges between nodes, instead of fixed nonlinearities at the nodes; the rival MLP and CNN use static activations such as Tanh, ReLU, and SiLU. The evaluation machinery is a three-class turn-intent prediction task built from simulated shank IMU signals, segmented into overlapping windows of 10, 20, or 30 samples (83, 167, or 250 ms) with 50% overlap, balanced by class weights in the loss, and optimized with Bayesian architecture search. Two paired comparisons—KAN versus MLP and FKAN versus CNN—isolate the effect of learnable activations; training on user-specific versus pooled data isolates the data-selection effect. Macro-averaged F1 is the outcome metric, and Wilcoxon signed-rank plus Bayesian paired t-tests give the significance claims.","core_discovery":"On the paper's own terms, learnable activation functions fail their promised test in this application. Per-participant Wilcoxon signed-rank comparisons between KAN and MLP gave p-values from 0.46 to 0.98, and between FKAN and CNN from 0.08 to 1.00; Bayesian paired t-tests across participants were also non-significant. For the training-data question, user-specific training significantly outperformed pooled training for MLP and KAN (p = 0.002 and p = 0.015), but not for CNN and FKAN (p = 0.127 and p = 0.833). The paper interprets these results as evidence that the three-label turn-prediction task is simple enough that static activations suffice, and that deep feature-extracting models can exploit pooled multi-participant data while conventional models still need individualization.","pith_inferences":["The paper's own explanation predicts that learnable activations would show an advantage on tasks with more action classes and larger data volumes; this is a direct, testable extension using a multi-activity IMU dataset.","If the pooled-training result for deep models repeats at larger participant counts, the personalization bottleneck in powered prosthetics shifts from data collection to model selection and on-device deployment.","The inference-time gap between KAN and MLP reported in the appendix suggests that realizing any future KAN advantage on a real controller will require approximating or pruning the B-spline or Jacobi basis evaluations, not just improving accuracy.","Recomputing the significance tests with non-overlapping folds or participant-level clustering would clarify how much of the MLP/KAN user-specific advantage is real, since the reported ten test divisions are overlapping subsamples of the same windows."],"forward_implications":["A VSTA-style controller could be trained on pooled multi-participant data if the deployed model is a CNN or FKAN, avoiding the cost of collecting per-user training data before first use.","For conventional models such as MLP or KAN, the paper's results predict a meaningful macro-F1 gain from retraining on the intended user's own data rather than relying on a population-trained model.","Adding KAN-style learnable activations to an existing MLP/CNN pipeline is unlikely to improve turn prediction, so the extra parameters and slower inference are not justified on this three-class task.","No single architecture wins for every participant; the best model varies, so a practical controller might select among CNN, FKAN, MLP, KAN, and the SVM baseline per user.","The roughly 300 ms swing-phase control window leaves room for CNN/FKAN prediction at tested PC-scale compute, though on-device latency still needs a microcontroller benchmark."],"supporting_citations":[{"why":"It supplies the baseline SVM turn-intent classifier and the prior finding that user-specific training beats pooled training for conventional models.","marker":"[9]"},{"why":"It introduces Kolmogorov-Arnold Networks with learnable B-spline activations, the method that Hypothesis 1 tests.","marker":"[31]"},{"why":"It introduces fractional Kolmogorov-Arnold Networks with trainable Jacobi basis functions, the deep learnable-activation comparator.","marker":"[33]"},{"why":"It provides the experimental protocol and IMU data from five amputee participants on which all models are evaluated.","marker":"[5]"},{"why":"It documents earlier KAN use for IMU-based activity recognition, motivating the open question about prosthesis control.","marker":"[37]"},{"why":"It supplies the automated machine-learning pipeline used as a strong baseline for comparison.","marker":"[47]"},{"why":"It supports the premise that deep learning improves with larger, more diverse data, which underpins the pooled-training expectation.","marker":"[39]"},{"why":"It justifies the 80/20 data splitting used to form training and test partitions.","marker":"[44]"},{"why":"It supplies the overlapping sliding-window segmentation used to convert IMU time series into classification inputs.","marker":"[45]"}],"fun_headline_variants":["KAN and FKAN no better than MLP and CNN for turn prediction","User-specific data beats pooled for ML models in prosthetic control","Deep nets handle pooled prosthetic data; shallow nets need personalization","KAN learnable activations fail to improve prosthetic turn control","KAN no better, but user-specific data still matters for ML"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The statistical conclusions assume the ten stratified divisions of each participant's test data are independent paired samples, even though they are overlapping subsamples of the same test windows, so the reported p-values rest on that independence assumption.","fun_headline_variants_meta":{"raw":{"variants":["KAN and FKAN no better than MLP and CNN for turn prediction","User-specific data beats pooled for ML models in prosthetic control","Deep nets handle pooled prosthetic data; shallow nets need personalization","KAN learnable activations fail to improve prosthetic turn control","KAN no better, but user-specific data still matters for ML"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000636,"raw_usage":{"total_tokens":2972,"prompt_tokens":1028,"completion_tokens":1944,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":1855}},"tokens_in":644,"tokens_out":1944,"duration_ms":12733,"temperature":1.0,"reasoning_tokens":1855,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:33:14.953338+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-evaluate the specific-versus-pooled comparisons on a single held-out test set per participant, or with block bootstrap over non-overlapping windows, so each observation is independent; if the MLP/KAN advantage over pooled training ceases to be significant at p < 0.05, the paper's Hypothesis 2 support would not survive.","supporting_citations":[{"cited_title":"Turn intent detection for control of a lower limb prosthesis,","cited_arxiv_id":null,"evidence_quote":"It supplies the baseline SVM turn-intent classifier and the prior finding that user-specific training beats pooled training for conventional models."},{"cited_title":"fkan: Fractional kolmogorov–arnold networks with trainable jacobi basis functions,","cited_arxiv_id":null,"evidence_quote":"It introduces fractional Kolmogorov-Arnold Networks with trainable Jacobi basis functions, the deep learnable-activation comparator."},{"cited_title":"Pilot testing of a variable stiffness transverse plane adapter for lower limb amputees,","cited_arxiv_id":null,"evidence_quote":"It provides the experimental protocol and IMU data from five amputee participants on which all models are evaluated."},{"cited_title":"Initial investigation of kolmogorov-arnold networks (kans) as feature extractors for imu based human activity recognition,","cited_arxiv_id":null,"evidence_quote":"It documents earlier KAN use for IMU-based activity recognition, motivating the open question about prosthesis control."},{"cited_title":"Efficient and robust automated machine learning,","cited_arxiv_id":null,"evidence_quote":"It supplies the automated machine-learning pipeline used as a strong baseline for comparison."},{"cited_title":"Bengio, I","cited_arxiv_id":null,"evidence_quote":"It supports the premise that deep learning improves with larger, more diverse data, which underpins the pooled-training expectation."},{"cited_title":"Split: An optimal method for data splitting,","cited_arxiv_id":null,"evidence_quote":"It justifies the 80/20 data splitting used to form training and test partitions."},{"cited_title":"Window size impact in human activity recognition,","cited_arxiv_id":null,"evidence_quote":"It supplies the overlapping sliding-window segmentation used to convert IMU time series into classification inputs."}],"review_version":1}