{"id":"1ba2f202-d533-45fe-bf92-b4ad778a3774","arxiv_id":"2412.17855","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Foxtsage, a population-based learning-rate controller combined with SGD, reports 42% lower aggregate loss than Adam on three benchmarks, but with only about 1% accuracy gains and a 330% time increase.","lead":"This paper introduces Foxtsage, a hybrid optimizer that wraps a nature-inspired population search around standard gradient descent to adjust the learning rate during neural network training. The authors compare Foxtsage with Adam on MNIST, CIFAR-10, and IMDB benchmarks, reporting lower training loss but roughly four times higher computation time.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3.2.3 pseudocode updates a single shared θ across candidates, so candidate losses are contaminated by prior updates; best_lr selection is invalid and the central Foxtsage-vs-Adam claim is unsupported.","rationale":"The reader's weakest_assumption—that the pseudocode's sequential updates contaminate candidate evaluation—is exactly the load-bearing concern I would raise. If this concern lands, the paper's central claim collapses: the learning-rate selection mechanism is not a fair comparison of learning rates, so the reported superiority of Foxtsage over Adam is unsubstantiated. The concern is internal to the paper: it follows directly from the pseudocode and the prose in Section 3.2.3. The proposed concrete test would settle it by comparing the as-written shared-θ implementation with an isolated per-candidate evaluation. I agree with the reader's REJECT verdict; the additional issues (Table 13 duplicate values, p-values comparing settings, compute-cost imbalance) reinforce rejection, but the sequential contamination is the single most load-bearing point. I am not raising a novel objection; I am confirming the reader's diagnosis and providing a direct experimental check. The paper has no formal verification or released code to counter this structural flaw, and the pseudocode is the only algorithmic specification. Hence no change to the reader's verdict is warranted.","tokens_in":21520,"tokens_out":3924,"duration_ms":32146,"concrete_test":"Re-implement the Section 3.2.3 pseudocode in PyTorch/TensorFlow and run two variants on MNIST logistic regression with Setting 1 (5 iterations, 10 agents). Variant A: exactly as written, using a single shared θ updated through the candidate loop. Variant B: before evaluating each candidate_lr, reset θ to a common checkpoint (e.g., the state at the start of the iteration) and train from that checkpoint for one epoch with that candidate; the final model is the one with the lowest per-candidate loss. Record the selected best_lr trajectory and final test loss for each variant. If Variant A and B select materially different learning rates or show different final losses (e.g., >5% relative difference), the shared-θ contamination is confirmed and the paper's reported results are not attributable to FOX-TSA's search.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Foxtsage dynamically adjusts the learning rate based on the best solution identified by the Hybrid FOX-TSA, and that this search mechanism yields a 42% loss reduction over Adam. The pseudocode in Section 3.2.3 makes this claim untestable as written. Lines 2.a.i–2.a.iv train a single model θ sequentially: for each candidate_lr in the population, the model is advanced by one epoch with that learning rate, and current_loss is computed on the already-updated θ. There is no re-initialization of θ between candidates, no per-candidate validation set, and no baseline snapshot. Consequently, current_loss for candidate j reflects the cumulative parameter updates made by candidates 1..j-1 in the current iteration and by all previous iterations. The loss cannot be attributed to candidate_lr alone, so the comparison that updates best_lr/best_loss is not a valid evaluation of learning-rate quality. The prose summary in Section 3.2.3 ('For each learning rate in the population, the model is trained for one epoch using SGD') compounds the ambiguity: if the implementation trains a fresh model per candidate, the pseudocode is misleading; if it follows the pseudocode, the search is confounded. Either way, the reported gains over Adam are not attributable to the FOX-TSA mechanism as described. This is the deepest flaw because it undermines the algorithm's internal validity before any comparison to Adam is made. Other issues (compute-matching, averaging losses of different magnitudes, Table 13 duplicates, and the mislabeled p-values) compound the problem, but the sequential contamination is the load-bearing one.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Foxtsage, a hybrid optimizer that wraps Stochastic Gradient Descent with a population-based learning-rate controller derived from the FOX-TSA metaheuristic, and benchmarks it against Adam on MNIST, CIFAR-10, and IMDB using MLPs, CNNs, and logistic regression. The central claim is that Foxtsage reduces mean loss by 42.03% and loss standard deviation by 42.19% relative to Adam, with modest accuracy/F1 gains, at roughly 3.3x higher compute cost, based on results aggregated over two settings (5 iterations with population 10, and 50 iterations with population 30).","tokens_in":21711,"tokens_out":2754,"duration_ms":25435,"significance":"If the central claim were sound, a population-based controller that dynamically schedules the learning rate for SGD and beats Adam on standard benchmarks would be a noteworthy empirical contribution to the optimizer literature. The paper has some strengths: it compares across multiple datasets and architectures, reports variability metrics (standard deviations), and is transparent about the large computational overhead. However, the experimental protocol as described does not support the headline claim: the algorithm's candidate evaluation step is confounded, the compute budget is not matched, and the aggregate Table 16 mixes losses of completely different scales. These issues are load-bearing rather than cosmetic, so the contribution as stated is not established.","major_comments":[{"comment":"The pseudocode's candidate evaluation is invalid. In the inner loop, for each candidate_lr, the same shared parameter vector θ is advanced with SGD for one epoch (θ ← θ - candidate_lr × ∇L(θ)), and then current_loss is computed on that already-updated θ. There is no re-initialization or per-candidate reset, so the loss attributed to candidate j already includes the cumulative updates from candidates 1 through j-1 in the same iteration and from all previous iterations. Consequently, the comparison that updates best_lr/best_loss does not isolate the effect of any single candidate learning rate. The prose summary ('For each learning rate in the population, the model is trained for one epoch using SGD') is ambiguous: if the implementation actually trains a fresh model per candidate, the pseudocode is misleading; if it follows the pseudocode, the selection of the best learning rate is confounded. Either way, the reported gains over Adam cannot be attributed to the FOX-TSA search mechanism as described.","section":"Section 3.2.3, pseudocode"},{"comment":"The comparison with Adam is not compute-matched. Foxtsage is run with 5 iterations × 10 candidates and 50 iterations × 30 candidates, which translates to 50 and 1500 SGD epoch-equivalents respectively, yet the number of epochs used for Adam is never reported. Loss values such as 99.45 (Table 5) and 231.36 (Table 9) for Foxtsage versus much lower values in other rows suggest that the two optimizers are not being compared after a comparable number of gradient steps. Without matching the number of epochs, iterations, or wall-clock budget, the claimed 42.03% loss reduction is not a meaningful comparison of optimizer quality.","section":"Section 3.4.2 and Section 4.1"},{"comment":"The aggregate loss metric in Table 16 is statistically meaningless because it averages losses across datasets that use different loss scales and different loss functions. For example, MNIST logistic regression reports loss means around 0.28-0.30, MNIST MLP reports 46.82-99.45, and CIFAR-10 CNN reports 0.12-231.36. Eq. (7) defines binary cross-entropy for binary classification, but the paper applies it to multi-class MNIST and CIFAR-10, and the reported values are far larger than a per-sample cross-entropy would be, suggesting either summed losses or an unnormalized objective. Averaging these heterogeneous quantities into a single 'Loss Mean' (16.402 vs 9.508) and computing a 42.03% improvement does not support the abstract's central quantitative claim.","section":"Table 16 and Eq. (7)"},{"comment":"Table 13 contains duplicated and likely incorrect entries: the rows for 'CNN (CIFAR-10)' and 'Logistic Regression (MNIST)' both report Loss Mean Improvement 41.34572907 and Accuracy Mean Improvement 0.747593951 in Setting 2, while the corresponding loss values in Table 14 (0.120085 vs 0.204734) yield a different percentage change. This inconsistency undermines the reliability of the reported percentage improvements and needs to be corrected and re-verified.","section":"Table 13 and Section 4.2.2"},{"comment":"The statistical significance reporting is not coherent. Table 15 is titled as comparing Setting 1 and Setting 2 ('between Setting 1 and Setting 2'), yet the text in Section 4.2.3 states that the p-values confirm that 'the performance differences between Foxtsage and Adam were significant.' No test name, sample size, or pairing structure is given, and the 'Statistic' column contains unexplained integers 4, 7, and 8. As written, the statistical tests do not support either claim reliably.","section":"Table 15 and Section 4.2.3"}],"minor_comments":[{"comment":"The paper contains numerous typos and grammatical errors, including 'optimisation' vs 'optimization' inconsistent usage, 'Hof Fman' in references, and the conclusion's 'precision 1.91' which should be '0.91%'.","section":"General"},{"comment":"Figure captions are incorrectly labeled: Figure 15 says 'Figure 13: CNN Training Loss on MNIST for Setting 2' and Figure 16 says 'Figure 14: CNN Validation Accuracy on MNIST for Setting 2', duplicating the earlier figure numbers.","section":"Figures 15-16"},{"comment":"The sentence 'Precision improved by 0.91% and recall by 1.02%, with their standard deviations improving by 7.95% and 7.87%' is unclear whether the standard deviation values are relative improvements or absolute changes.","section":"Section 4.2.4"},{"comment":"Several references lack complete bibliographic information, such as page numbers or venue details (e.g., reference 26 for Kingma and Ba, reference 9 for Defossez et al.), and some citations are repeated with inconsistent author names (e.g., 'Aula, S.A.' vs 'Sirwan A. Aula').","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper's central claim is not supported by the experimental protocol as described. The shared-θ contamination in Section 3.2.3 is a fundamental flaw that invalidates the learning-rate selection, and the aggregate loss metric in Table 16 is an arithmetic of incompatible quantities. These are not local issues that a small revision could repair; they require a redesign of the evaluation. The manuscript would also benefit from code release and a more rigorous, compute-matched comparison protocol. I see no conflict of interest, but note that the authors have multiple self-citations, which is not problematic per se but should be checked for relevance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a quick look only as a cautionary example. The core idea is straightforward: use a FOX-TSA population to adjust SGD's learning rate. That is not new — population-based hyperparameter search is well established — but the paper does ship a concrete algorithm, runs it across several standard datasets, and honestly reports the 330% time increase. Good for transparency, and the citation pattern is mostly appropriate, leaning on their own FOX-TSA work, which is expected given they are extending it.\n\nThe problem is the protocol. The pseudocode in Section 3.2.3 updates a single shared θ sequentially for each candidate_lr, so the loss computed for candidate j already includes the parameter updates from candidates 1 through j-1. There is no re-initialization, no per-candidate validation set, no baseline snapshot. The best_lr selection is therefore not a valid comparison of learning rates. This is the load-bearing flaw, and the stress-test has it right.\n\nBeyond that, the Adam comparison is not compute-matched: Foxtsage gets P times I SGD epochs, while Adam's epoch count is never reported. The headline 42% loss reduction is an average of per-dataset loss means that span orders of magnitude (0.29 to 231), so it's an artifact of the averaging. The p-values in Table 15 compare Setting 1 vs Setting 2, not Foxtsage vs Adam, despite the abstract implying significance tests for the optimizer. Table 13 has duplicate values for two different datasets, and the text contradicts Table 7 on the MNIST CNN loss improvement. The accuracy gains are under 1%, so even at face value this is a marginal result at four times the compute.\n\nThe paper is not a fraud; it is just not ready. The authors seem to have run real experiments and reported real numbers, but the description of what they did is too ambiguous to be reproducible, and the claims outrun the evidence. I would not send this to peer review in its current form. A desk reject is appropriate. If the authors fix the protocol, report epoch counts, and redo the statistics, there might be a modest empirical note in it.","headline":"A population-based learning-rate search wrapped around SGD, but the comparison with Adam is invalidated by a confounded evaluation protocol and unreported compute budgets.","tokens_in":22425,"tokens_out":2883,"would_cite":false,"duration_ms":25205,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that Foxtsage, a FOX-TSA-guided SGD, reduces training loss by 42.03% versus Adam across three benchmark datasets, at 4.3x the compute time.","keywords":["Foxtsage","FOX-TSA","Stochastic Gradient Descent","Adam optimizer","learning rate adaptation","neural network optimization","hybrid metaheuristic","benchmark comparison"],"falsifier":"Rerun the Foxtsage pseudocode with the model parameters reset to a common random initialization before each candidate learning rate is evaluated, or with each candidate training a fresh copy of the model; if the 42% loss advantage over Adam disappears, the claimed benefit is an artifact of the sequential update rather than of learning-rate search.","tokens_in":21151,"feed_emoji":"🦊","tokens_out":8411,"duration_ms":57133,"temperature":0.7,"pith_summary":"This paper tries to establish that a hybrid optimizer called Foxtsage—SGD with its learning rate steered by the FOX-TSA population search—trains neural networks better than the widely used Adam optimizer. On MNIST, CIFAR-10, and IMDB, across logistic regression, MLP, and CNN models, the authors report a 42.03% lower mean training loss, a 42.19% lower loss standard deviation, and modest gains in accuracy, precision, recall, and F1-score, at a 330.87% higher mean computation time. If true, the result would show that a population-based learning-rate controller can beat Adam's adaptive moments on small-scale networks, though only when wall-clock time is not the limiting factor. The claim rests on the Section 3.2.3 pseudocode, in which each candidate learning rate is evaluated by one epoch of SGD on a single shared model before FOX-TSA updates the population around the best loss.","feed_headline":"Hybrid optimizer cuts neural-net loss 42% vs Adam","feed_subtitle":"Foxtsage wraps FOX-TSA around SGD to tune the learning rate, but uses 4.3x more compute.","key_machinery":"The central mechanism is the Foxtsage optimizer, a population-based learning-rate controller wrapped around SGD. A population of candidate learning rates is initialized uniformly in $[\\eta_{\\min}, \\eta_{\\max}]$; each candidate trains the model for one epoch via $\\theta \\leftarrow \\theta - \\mathrm{candidate\\_lr} \\times \\nabla L(\\theta)$, the resulting loss is compared to the current best, and FOX-TSA then perturbs the population around the best learning rate using an exploration move (multiply by a Gaussian perturbation when $r<0.5$) or an exploitation move (divide by a Gaussian perturbation when $r\\ge 0.5$), clipping back into bounds. The load-bearing identity is Eq. (6), $\\eta_t = \\eta_{\\mathrm{base}}/(1 + \\alpha f_{\\mathrm{best}}(t))$, which ties the SGD step size to the best fitness found by FOX-TSA; the pseudocode evaluates candidates sequentially on one shared model, so the 'best' learning rate is selected on a trajectory that already contains earlier candidates' updates.","core_discovery":"Foxtsage is presented as an adaptive learning-rate design: at each iteration, the hybrid FOX-TSA algorithm searches a population of candidate learning rates, and the best one found so far sets the SGD step via $\\eta_t = \\eta_{\\mathrm{base}}/(1 + \\alpha f_{\\mathrm{best}}(t))$. The paper's headline result, aggregated in its Table 16, is that Foxtsage reduces mean training loss by 42.03% (9.508 vs. 16.402) and loss standard deviation by 42.19% (20.86 vs. 36.085) relative to Adam, with accuracy mean up 0.78%, precision up 0.91%, recall up 1.02%, and F1-score up 0.89%. The same table reports a 330.87% increase in mean computation time, which the authors acknowledge as the main trade-off. The authors interpret the lower loss and smaller variance as evidence that Foxtsage is a robust alternative to Adam for neural-network training, and they argue that the FOX-TSA exploration/exploitation balance is what lets SGD escape the pitfalls of a static learning rate.","pith_inferences":["If Foxtsage's loss gains largely reflect the extra gradient evaluations spent evaluating a population, then a fairer baseline is Adam run for the same number of epochs or the same wall-clock time; the paper's own 330% time overhead makes this a live possibility.","The sequential update in Section 3.2.3 effectively turns the 'population' into a single trajectory with a time-varying learning rate; testing Foxtsage with population size 1 would reveal how much of the benefit comes from FOX-TSA search versus an implicit learning-rate schedule.","A direct comparison to Adam with cosine annealing or other standard learning-rate schedules would clarify whether Foxtsage's advantage is specific to the FOX-TSA mechanism or just a form of adaptive decay.","On CIFAR-10 Setting 2, Foxtsage's accuracy mean (0.7623) is slightly below Adam's (0.7681) despite a lower loss mean, so the claimed superiority is loss-driven; practitioners should check whether the loss metric aligns with their deployment metric before adopting Foxtsage."],"forward_implications":["Foxtsage reduces mean training loss by 42.03% relative to Adam across the benchmark settings, with a 42.19% reduction in loss standard deviation.","Accuracy, precision, recall, and F1-score improve on average by 0.78%, 0.91%, 1.02%, and 0.89%, respectively.","The loss improvements grow with the computational budget: Setting 2 (50 iterations, population 30) yields larger reductions than Setting 1 (5 iterations, population 10).","These gains come at a 330.87% increase in mean computation time and a 266.13% increase in its standard deviation, so Foxtsage is not a drop-in replacement under tight time budgets.","The authors report statistically significant differences between the two settings (p = 0.0137 for loss mean, p = 0.0371 for accuracy mean, p = 0.0488 for F1-score) and interpret Foxtsage as a robust alternative to Adam."],"supporting_citations":[{"why":"Supplies the Adam optimizer that Foxtsage is benchmarked against, including its update equations and standard hyperparameters.","marker":"Kingma and Ba 2014"},{"why":"Provides the FOX algorithm whose exploration dynamics form the global-search half of the FOX-TSA learning-rate controller.","marker":"Mohammed and Rashid 2023"},{"why":"Provides the Tree-Seed Algorithm that contributes the exploitation mechanism for fine-tuning the best learning rate.","marker":"Kiran 2015"},{"why":"Defines the hybrid FOX-TSA algorithm that Foxtsage wraps around SGD; the paper cites it as the source of the search mechanism.","marker":"S.A. Aula and Rashid 2024"},{"why":"Earlier application of FOX-TSA to MLP training and tourism data, cited as motivation that the hybrid can improve neural-network optimization.","marker":"Sirwan A. Aula and Rashid 2024"}],"fun_headline_variants":["Foxtsage cuts neural-net loss 42% vs Adam, but takes 4x longer","Hybrid optimizer Foxtsage: 42% lower loss, 330% more compute","Foxtsage beats Adam: loss down 42%, but time up 4x","Foxtsage: 42% loss reduction, but 4.3x slower than Adam","FOX-TSA + SGD cuts neural-net loss 42% over Adam, at 4x cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each candidate learning rate is evaluated fairly, but the pseudocode updates a single shared model sequentially for all candidates, so later candidates inherit parameter changes from earlier ones and the reported 'best' learning rate may simply be the last one tried.","fun_headline_variants_meta":{"raw":{"variants":["Foxtsage cuts neural-net loss 42% vs Adam, but takes 4x longer","Hybrid optimizer Foxtsage: 42% lower loss, 330% more compute","Foxtsage beats Adam: loss down 42%, but time up 4x","Foxtsage: 42% loss reduction, but 4.3x slower than Adam","FOX-TSA + SGD cuts neural-net loss 42% over Adam, at 4x cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001462,"raw_usage":{"total_tokens":5933,"prompt_tokens":1045,"completion_tokens":4888,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":661,"completion_tokens_details":{"reasoning_tokens":4768}},"tokens_in":661,"tokens_out":4888,"duration_ms":26051,"temperature":1.0,"reasoning_tokens":4768,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:21:10.211028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the Foxtsage pseudocode with the model parameters reset to a common random initialization before each candidate learning rate is evaluated, or with each candidate training a fresh copy of the model; if the 42% loss advantage over Adam disappears, the claimed benefit is an artifact of the sequential update rather than of learning-rate search.","supporting_citations":[{"cited_title":"FOX: A FOX -Inspired Optimization Algorithm","cited_arxiv_id":null,"evidence_quote":"Provides the FOX algorithm whose exploration dynamics form the global-search half of the FOX-TSA learning-rate controller."},{"cited_title":"TSA: Tree -Seed Algorithm for Continuous Optimization","cited_arxiv_id":null,"evidence_quote":"Provides the Tree-Seed Algorithm that contributes the exploitation mechanism for fine-tuning the best learning rate."}],"review_version":1}